Pixel GPU Online DQM Client changes for CMSHLT-3147

[mmusich]

PR description:

This PR is opened in preparation of the integration of the ticket CMSHLT-3147.
In a nutshell that ticket suggests that the DQM plugins which are currently in the Path DQM_PixelReconstruction_v (Sequence: HLTDQMPixelReconstruction) and in the Path DQM_PixelReconstruction_v (Sequece HLTDQMPixelReconstruction) can be removed from the HLT combined table, as long as the Tracker-DPG and/or Tracking-POG experts move the corresponding monitoring sequence into the appropriate online-DQM client (which reads the DQMGPUvsCPU streamer files).
This latter point is proposed in this PR, in which we change the client DQM/Integration/python/clients/pixelgpu_dqm_sourceclient-live_cfg.py to include directly the modules responsible of monitoring of the GPU vs CPU comparisons, as the input data to feed them will be sent by the input DQMGPUvsCPU stream.

PR validation:

I created a suitable set of streamer files using the following script:

#!/bin/bash -ex
RUNNUMBER=397813
LUMISECTION=131

# cmsrel CMSSW_15_1_X_2025-10-22-2300
# cd CMSSW_15_1_X_2025-10-22-2300/src/
# cmsenv
# git cms-addpkg DataFormats/TrackSoA DataFormats/TrackingRecHitSoA DQM/Integration
# git remote add AdrianoDee git@github.com:AdrianoDee/cmssw.git; git fetch AdrianoDee
# git cherry-pick 1eb24e6536182945d6f7ac55fedc023a9f0d4410
# git cherry-pick 9a3ef9ae14f452aa15d3f868b97c63f6b996b1dc
# cmsenv
# scram b

INPUTFILE=root://eoscms.cern.ch//eos/cms/store/express/Run2025F/ExpressPhysics/FEVT/Express-v2/000/397/813/00000/f95a79f9-18ef-48f1-951a-2ae78c1a107f.root
rm -rf run${RUNNUMBER}*

# run on 100 events of LS 131, with 100 events per input file
convertToRaw -f 100 -l 100 -r ${RUNNUMBER}:${LUMISECTION} -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
hltConfigFromDB --configName /users/musich/tests/dev/CMSSW_15_1_0/CMSHLT-3147/GRun > "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load("run${RUNNUMBER}_cff")

# to run without any HLT prescales
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
process.GlobalTag.globaltag = cms.string( "150X_dataRun3_HLT_v1" )
# # to run using the same HLT prescales as used online in LS 1000
# process.PrescaleService.forceDefault = True

from HLTrigger.Configuration.common import *
def customizeHLTFor49149(process):
    ca_producers_pp = ['CAHitNtupletAlpakaPhase1@alpaka','alpaka_serial_sync::CAHitNtupletAlpakaPhase1']
    for ca_producer in ca_producers_pp:
        for prod in producers_by_type(process, ca_producer):
            if hasattr(prod, 'geometry'):
                g = getattr(prod, 'geometry')
                g.startingPairs = cms.vuint32( [i for i in range(8)] + [i for i in range(13,19)])
                setattr(prod, 'geometry', g)
    return process

process = customizeHLTFor49149(process)
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

bash -c 'echo $$ > cmsrun.pid; exec cmsRun hlt.py &> hlt.log'
job_pid=$(cat cmsrun.pid)
echo "cmsRun is running with PID: $job_pid"

# remove input files to save space
rm -f run397813/run397813_ls0*_index*.*

# prepare the files by concatenating the .ini and .dat files
mkdir -p prepared
cat run397813/run397813_ls0000_streamDQMGPUvsCPU_pid${job_pid}.ini run397813/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.dat > prepared/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.dat
cp run397813/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.jsn prepared/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.jsn
# manually edit to remove the extra 0 in the json file.

and then placed the resulting files under the path $CMSSW_BASE/src/DQM/Integration/data/run397813.
The resulting streamers files have been committed to the cms-data DQM-Integration repository via PR cms-data/DQM-Integration#14 (that needs to be tested together with this one).
I then run the unit test of the package via:

 scram b runtests_TestDQMOnlineClient-pixelgpu_dqm_sourceclient
 scram b runtests_TestDQMOnlineClient-hcalgpu_dqm_sourceclient
 scram b runtests_TestDQMOnlineClient-ecalgpu_dqm_sourceclient
 scram b runtests_TestDQMOnlineClient-pfgpu_dqm_sourceclient

and I did not observe any issues.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, but it will be backported to CMSSW_15_1_X for 2025 datat-taking operations.

[cmsbuild]

cms-bot internal usage

[mmusich]

test parameters:

• pull_request = Reorganization due to CMSHLT-3147 cms-data/DQM-Integration#14

[mmusich]

@cmsbuild, please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49237/46584

[cmsbuild]

A new Pull Request was created by @mmusich for master.

It involves the following packages:

• DQM/Integration (dqm)

@ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@batinkov, @francescobrivio, @threus this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f627f4/48873/summary.html
COMMIT: 1a6f4ee
CMSSW: CMSSW_16_0_X_2025-10-28-1700/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49237/48873/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3939953
• DQMHistoTests: Total failures: 69
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3939864
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 188 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[gabrielmscampos]

• P5 tests OK using 16_0_0_pre1 and streamers from Reorganization due to CMSHLT-3147 cms-data/DQM-Integration#14

Edit:

• We are going to perform more tests to measure the impact on the storage usage due to the event size increase

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-data/DQM-Integration#14

[mmusich]

@gabrielmscampos

We are going to perform more tests to measure the impact on the storage usage due to the event size increase

do yo have news concerning this point?
Thank you

[nothingface0]

do yo have news concerning this point? Thank you

We are running some tests now, hopefully within the day we'll have a better picture.

[mmusich]

We are running some tests now, hopefully within the day we'll have a better picture.

given CMSHLT-3147 has converged, can this and the corresponding cms-data PR be signed?

[nothingface0]

+dqm

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-data/DQM-Integration#14

[ftenchini]

+1