Implement GPU vs CPU comparison for HLT pixel tracking heterogeneous products in patatrack workflows

[mmusich]

PR description:

This PR is a quick follow-up upon #49105.
Thanks to the recently merged PR #49303 we are now saving into the HLTDebugRAW and HLTDebugFEVT event content definitions also hltSiPixelRecHitsSoA*, hltPixelTracksSoA* and hltPixelVerticesSoA*.
This puts us in a position to include in the HLTHeterogeneousMonitoringSequence defined in #49105 all the various monitoring modules for the pixel rechits, tracks and vertex SoAs. This is carried out in commit e8814ac.
I profit of this PR to fix some typos / oddities in commit eaef3cd as well as adding a harvesting step for the @hltGPUvsCPU sequence in commit 0e98d60.

PR validation:

Run successfully:

runTheMatrix.py --what gpu -l 17034.402 -t 4 -j 8 --ibeos

and checked that the desired output plots are available (see for example here).

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, it won't be backported.

[mmusich]

enable gpu

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49340/46712

• There are other open Pull requests which might conflict with changes you have proposed:
  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): [mkFit] final fit initial PR and testing workflow #49339
  • File DQM/SiPixelHeterogeneous/plugins/SiPixelCompareRecHitsSoA.cc modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction  #45853
  • File DQMOffline/Configuration/python/DQMOffline_SecondStep_cff.py modified in PR(s): Tau DQM miniAOD validation for DATA #44029, Update BTV offline DQM sequences #46838

[mmusich]

@cmsbuild, please test

[cmsbuild]

A new Pull Request was created by @mmusich for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv)
• DQM/SiPixelHeterogeneous (dqm)
• DQMOffline/Configuration (dqm)
• DQMOffline/Trigger (dqm)

@AdrianoDee, @DickyChant, @antoniovagnerini, @ctarricone, @gabrielmscampos, @miquork, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@Fedespring, @HuguesBrun, @Martin-Grunewald, @cericeci, @fabiocos, @fioriNTU, @idebruyn, @jandrea, @jhgoh, @makortel, @missirol, @mtosi, @rociovilar, @slomeo, @threus, @trocino this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+1

Size: This PR adds an extra 56KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-931520/49323/summary.html
COMMIT: 0e98d60
CMSSW: CMSSW_16_0_X_2025-11-06-1400/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49340/49323/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 3 lines to the logs
• Reco comparison results: 4 differences found in the comparisons
• Reco comparison had 2 failed jobs
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3939953
• DQMHistoTests: Total failures: 30
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3939903
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 188 edm output root files, 51 DQM output files
• TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

• You potentially added 73 lines to the logs
• Reco comparison results: 205 differences found in the comparisons
• Reco comparison had 6 failed jobs
• DQMHistoTests: Total files compared: 11
• DQMHistoTests: Total histograms compared: 147869
• DQMHistoTests: Total failures: 30949
• DQMHistoTests: Total nulls: 11
• DQMHistoTests: Total successes: 116909
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 125189.87399999998 KiB( 10 files compared)
• DQMHistoSizes: changed ( 17034.402,... ): 20864.979 KiB HLT/HeterogeneousComparisons
• Checked 42 log files, 45 edm output root files, 11 DQM output files
• TriggerResults: found differences in 2 / 10 workflows

AMD_W7900 Comparison Summary

Summary:

• You potentially added 73 lines to the logs
• Reco comparison results: 198 differences found in the comparisons
• Reco comparison had 6 failed jobs
• DQMHistoTests: Total files compared: 11
• DQMHistoTests: Total histograms compared: 147869
• DQMHistoTests: Total failures: 31978
• DQMHistoTests: Total nulls: 8
• DQMHistoTests: Total successes: 115883
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 125189.87399999998 KiB( 10 files compared)
• DQMHistoSizes: changed ( 17034.402,... ): 20864.979 KiB HLT/HeterogeneousComparisons
• Checked 42 log files, 45 edm output root files, 11 DQM output files
• TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

• You potentially added 82 lines to the logs
• Reco comparison results: 206 differences found in the comparisons
• Reco comparison had 6 failed jobs
• DQMHistoTests: Total files compared: 11
• DQMHistoTests: Total histograms compared: 147869
• DQMHistoTests: Total failures: 29836
• DQMHistoTests: Total nulls: 13
• DQMHistoTests: Total successes: 118020
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 125189.87399999998 KiB( 10 files compared)
• DQMHistoSizes: changed ( 17034.402,... ): 20864.979 KiB HLT/HeterogeneousComparisons
• Checked 42 log files, 45 edm output root files, 11 DQM output files
• TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

• You potentially added 99 lines to the logs
• Reco comparison results: 243 differences found in the comparisons
• Reco comparison had 6 failed jobs
• DQMHistoTests: Total files compared: 11
• DQMHistoTests: Total histograms compared: 147869
• DQMHistoTests: Total failures: 25897
• DQMHistoTests: Total nulls: 9
• DQMHistoTests: Total successes: 121963
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 125189.87399999998 KiB( 10 files compared)
• DQMHistoSizes: changed ( 17034.402,... ): 20864.979 KiB HLT/HeterogeneousComparisons
• Checked 42 log files, 45 edm output root files, 11 DQM output files
• TriggerResults: found differences in 1 / 10 workflows

NVIDIA_T4 Comparison Summary

Summary:

• You potentially added 85 lines to the logs
• Reco comparison results: 257 differences found in the comparisons
• Reco comparison had 6 failed jobs
• DQMHistoTests: Total files compared: 11
• DQMHistoTests: Total histograms compared: 147869
• DQMHistoTests: Total failures: 26191
• DQMHistoTests: Total nulls: 12
• DQMHistoTests: Total successes: 121666
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 125189.87399999998 KiB( 10 files compared)
• DQMHistoSizes: changed ( 17034.402,... ): 20864.979 KiB HLT/HeterogeneousComparisons
• Checked 42 log files, 45 edm output root files, 11 DQM output files
• TriggerResults: found differences in 3 / 10 workflows

[gabrielmscampos]

+dqm

[AdrianoDee]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1