TRK CPU vs. GPU path and infrastructure for Phase 2 HLT by VourMa · Pull Request #50336 · cms-sw/cmssw

VourMa · 2026-03-06T17:38:07Z

This PR introduces CPU vs. GPU validation for the Phase 2 HLT, implementing the comparison for the tracking sequence. The relevant workflow is 0.7503.

The approach has been presented at a recent HLT Upgrade meeting, and has been updated according to the feedback received there, i.e.:

No new menu: the path is added to the "regular" menu;
No new validation sequence: A DQM sequence is run instead.
- Due to issues with the DQM step (see Duplicate definition of simEcalTriggerPrimitiveDigis in different files? #48976), the DQM sequence has been included within the HLT menu.
Instead of the hltInitialStepTracks, tracks produced directly from the seeds given to the building algorithm are used;
Monitoring of the pixel tracks SoA-s and the harvesting step for the track-to-track comparisons have been added;
The procModifier has been generalized to apply to all HLT heterogeneous comparisons, modifying all reconstruction sequences to add the serial sync modules.

Here is the new DQM output (click to show):

cmsbuild · 2026-03-06T17:38:36Z

cms-bot internal usage

cmsbuild · 2026-03-06T17:40:06Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50336/48426

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/EventContent/python/EventContent_cff.py modified in PR(s): TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File Configuration/PyReleaseValidation/README.md modified in PR(s): TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File Configuration/PyReleaseValidation/python/relval_Run4.py modified in PR(s): TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): update phase-2 HLT timing script and use alpaka modifier in NGT workflows #49821, TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932, Particle Flow cluster validation in ECAL #50292, MLPF Run 3 inference code and model update #50316
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): Phase2 Single_Tau_Trigger Path Added #49637, TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File HLTriggerOffline/Common/python/HLTValidation_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292

cmsbuild · 2026-03-06T17:40:29Z

A new Pull Request was created by @VourMa for master.

It involves the following packages:

Configuration/EventContent (operations)
Configuration/ProcessModifiers (operations)
Configuration/PyReleaseValidation (pdmv)
DQM/TrackingMonitorSource (dqm)
HLTrigger/Configuration (hlt)
HLTriggerOffline/Common (dqm)
Validation/Configuration (dqm, simulation)
Validation/RecoTrack (dqm)

@AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @civanch, @cmsbuild, @ctarricone, @davidlange6, @fabiocos, @ftenchini, @gabrielmscampos, @kpedro88, @mandrenguyen, @mdhildreth, @miquork, @mmusich, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @Martin-Grunewald, @SohamBhattacharya, @VinInn, @VourMa, @apsallid, @arossi83, @denizsun, @dgulhan, @elusian, @fabiocos, @felicepantaleo, @fioriNTU, @idebruyn, @jandrea, @makortel, @missirol, @mmasciov, @mmusich, @mtosi, @richa2710, @rovere, @salimcerci, @slomeo, @sroychow, @threus, @wmtford this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2026-03-09T07:58:57Z

test parameters:

enable_tests = gpu, hlt_p2_integration, hlt_p2_timing
workflows_gpu = 34434.7503
workflows = ph2_hlt
relvals_opt = -w upgrade,standard
relvals_opt_gpu = -w upgrade,standard

mmusich · 2026-03-08T23:00:10Z

 )

+from DQM.TrackingMonitorSource.TrackToTrackComparisonHists_cfi import TrackToTrackComparisonHists as _TrackToTrackComparisonHists
+hltPixelTrackToTrackSerialSync = _TrackToTrackComparisonHists.clone(


as this uses only "data" and not simultation truth, I would naively assume it should go into "DQM" rather than "Validation"

Sure, I can put it in a more appropriate file. Do you have any proposal about where to move it?

I would create a new file in DQM/TrackingMonitorSource/python and reference it in the phase-2 hlt DQM sequence

@VourMa

While we wait for a resolution of #48976 that would allow to run the DQM step in the same job as HLT, I think an acceptable compromise is to run the DQM modules themselves within the HLT menu as we did for most of Run 3 for the pixel heterogeneous tracking.
Here is a commit to that effect mmusich@07693a7.
I profited to:

add the monitoring of the pixel tracks SoA-s

add the harvesting step for the track-to-track comparisons

Tested with runTheMatrix.py --what upgrade -l 34434.7503 -t 4 -j 8 on lxplus-gpu

Thanks for this, @mmusich! I will take a look and included today/tomorrow so that we can move forward.

I applied the commit verbatim

mmusich · 2026-03-09T12:12:01Z

+                                outputCommands = FEVTDEBUGHLTEventContent.outputCommands+[
+                                    'keep *_hltInitialStepTrajectorySeedsLSTTracks_*_*',
+                                    'keep *_hltInitialStepTrajectorySeedsLSTTracksSerialSync_*_*',
+                                    'keep *_hltPhase2PixelTracks_*_*',


some of these are already saved by default please clean up the redundant keep statements.

cmssw/Configuration/EventContent/python/EventContent_cff.py

Lines 709 to 720 in 7e37aba

phase2_tracker.toModify(FEVTDEBUGHLTEventContent,

outputCommands = FEVTDEBUGHLTEventContent.outputCommands+[

'keep *_hltSiPixelClusters_*_*',

'keep *_hltSiPhase2Clusters_*_*',

'keep *_hltPhase2PixelTracks_*_*',

'keep *_hltPhase2PixelVertices_*_*',

'keep *_hltGeneralTracks_*_*',

'keep *_hltInitialStepTrackSelectionHighPurity_*_*',

'keep *_hltHighPtTripletStepTrackSelectionHighPurity_*_*',

'keep *_hltInitialStepTracksT4T5TCLST_*_*',

'keep *_hltOfflinePrimaryVertices_*_*',

])

I updated the file accordingly.

cmsbuild · 2026-03-16T19:54:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50336/48552

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/EventContent/python/EventContent_cff.py modified in PR(s): TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File Configuration/PyReleaseValidation/README.md modified in PR(s): TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File Configuration/PyReleaseValidation/python/relval_Run4.py modified in PR(s): TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932, Introduce Phase 2 FastSim (tracker) workflows #50437
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): update phase-2 HLT timing script and use alpaka modifier in NGT workflows #49821, TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932, Particle Flow cluster validation in ECAL #50292, MLPF Run 3 inference code and model update #50316, Introduce Phase 2 FastSim (tracker) workflows #50437
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): Phase2 Single_Tau_Trigger Path Added #49637, TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File HLTriggerOffline/Common/python/HLTValidation_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292

cmsbuild · 2026-03-16T19:55:08Z

Pull request #50336 was updated. @AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @civanch, @cmsbuild, @ctarricone, @davidlange6, @fabiocos, @ftenchini, @gabrielmscampos, @kpedro88, @mandrenguyen, @mdhildreth, @miquork, @mmusich, @nothingface0, @rseidita can you please check and sign again.

mmusich · 2026-03-16T19:57:24Z

@cmsbuild, please test

cmsbuild · 2026-03-18T18:45:11Z

+1

Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dfc9e5/52071/summary.html
COMMIT: 2cd1f9b
CMSSW: CMSSW_16_1_X_2026-03-17-2300/el8_amd64_gcc13
Additional Tests: GPU,HLT_P2_INTEGRATION,HLT_P2_TIMING,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50336/52071/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 19 differences found in the comparisons
DQMHistoTests: Total files compared: 68
DQMHistoTests: Total histograms compared: 4800317
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4800285
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 67 files compared)
Checked 282 log files, 244 edm output root files, 68 DQM output files
TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

You potentially removed 27 lines from the logs
Reco comparison results: 369 differences found in the comparisons
DQMHistoTests: Total files compared: 12
DQMHistoTests: Total histograms compared: 184058
DQMHistoTests: Total failures: 29147
DQMHistoTests: Total nulls: 17
DQMHistoTests: Total successes: 154894
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
Checked 46 log files, 48 edm output root files, 12 DQM output files
TriggerResults: found differences in 1 / 11 workflows

AMD_W7900 Comparison Summary

Summary:

You potentially added 19 lines to the logs
Reco comparison results: 356 differences found in the comparisons
DQMHistoTests: Total files compared: 12
DQMHistoTests: Total histograms compared: 184058
DQMHistoTests: Total failures: 29309
DQMHistoTests: Total nulls: 13
DQMHistoTests: Total successes: 154736
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
Checked 46 log files, 48 edm output root files, 12 DQM output files
TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

You potentially removed 20 lines from the logs
Reco comparison results: 367 differences found in the comparisons
DQMHistoTests: Total files compared: 12
DQMHistoTests: Total histograms compared: 184058
DQMHistoTests: Total failures: 25359
DQMHistoTests: Total nulls: 18
DQMHistoTests: Total successes: 158681
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
Checked 46 log files, 48 edm output root files, 12 DQM output files
TriggerResults: found differences in 1 / 11 workflows

NVIDIA_L40S Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 377 differences found in the comparisons
DQMHistoTests: Total files compared: 12
DQMHistoTests: Total histograms compared: 184058
DQMHistoTests: Total failures: 27164
DQMHistoTests: Total nulls: 17
DQMHistoTests: Total successes: 156877
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
Checked 46 log files, 48 edm output root files, 12 DQM output files
TriggerResults: found differences in 2 / 11 workflows

mmusich · 2026-03-19T07:27:00Z

workflows_gpu = 34434.7503

It looks like this workflow did run in the GPU matrix: logs, but I can't see it in the bin-by-bin comparisons.

To maximize usefulness I would suggest to move the workflow from the upgrade to the gpu matrix and then put it in here in the cms-bot configuration. This can be done at a later time.

mmusich · 2026-03-19T07:28:06Z

+hlt

for SegmentLinking@2cd1f9b
PR according to description (@VourMa you might want to edit it to reflect the current status of the PR) and follow-up review
there is still some opportunity to make the workflow more useful for PR checks (see TRK CPU vs. GPU path and infrastructure for Phase 2 HLT #50336 (comment))

VourMa · 2026-03-19T09:00:15Z

To maximize usefulness I would suggest to move the workflow from the upgrade to the gpu matrix and then put it in here in the cms-bot configuration. This can be done at a later time.

Yes, since this requires an external, let's do it in a separate (set of) PR(s). I will prepare it and submit when this is in.

PR according to description (@VourMa you might want to edit it to reflect the current status of the PR) and follow-up review

Right, I will do so later today.

civanch · 2026-03-19T14:11:01Z

+1

gabrielmscampos · 2026-03-19T15:08:06Z

+dqm

mmusich · 2026-03-20T09:22:28Z

@cms-sw/pdmv-l2 kind ping

AdrianoDee · 2026-03-23T13:34:04Z

+pdmv

cmsbuild · 2026-03-23T13:34:26Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

ftenchini · 2026-03-23T16:09:05Z

+1

cmsbuild added this to the CMSSW_16_1_X milestone Mar 6, 2026

cmsbuild added simulation-pending dqm-pending hlt-pending operations-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending tracking labels Mar 6, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 6, 2026

mmusich reviewed Mar 9, 2026

View reviewed changes

This was referenced Mar 13, 2026

Phase2 Single_Tau_Trigger Path Added #49637

Open

TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932

Open

Introduce Phase 2 FastSim (tracker) workflows #50437

Merged

VourMa force-pushed the GPUValidationForPhase2HLTTRK branch from ae0972b to 846424d Compare March 16, 2026 19:52

cmsbuild added code-checks-pending and removed code-checks-approved labels Mar 16, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 16, 2026

cmsbuild added operations-approved tests-approved and removed operations-pending tests-started labels Mar 18, 2026

cmsbuild added hlt-approved and removed hlt-pending labels Mar 19, 2026

cmsbuild added simulation-approved and removed simulation-pending labels Mar 19, 2026

cmsbuild added dqm-approved and removed dqm-pending labels Mar 19, 2026

cmsbuild mentioned this pull request Mar 19, 2026

Particle Flow cluster validation in ECAL #50292

Open

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Mar 23, 2026

cmsbuild added orp-approved and removed orp-pending labels Mar 23, 2026

cmsbuild merged commit 5b39379 into cms-sw:master Mar 23, 2026
25 checks passed

This was referenced Mar 24, 2026

Add Phase 2 HLT CPU vs. GPU comparison wf in default GPU tests cms-sw/cms-bot#2710

Merged

Add Phase 2 HLT CPU vs. GPU wf to the ph2_hlt PR tests #50505

Open

	phase2_tracker.toModify(FEVTDEBUGHLTEventContent,
	outputCommands = FEVTDEBUGHLTEventContent.outputCommands+[
	'keep _hltSiPixelClusters__*',
	'keep _hltSiPhase2Clusters__*',
	'keep _hltPhase2PixelTracks__*',
	'keep _hltPhase2PixelVertices__*',
	'keep _hltGeneralTracks__*',
	'keep _hltInitialStepTrackSelectionHighPurity__*',
	'keep _hltHighPtTripletStepTrackSelectionHighPurity__*',
	'keep _hltInitialStepTracksT4T5TCLST__*',
	'keep _hltOfflinePrimaryVertices__*',
	])

Conversation

VourMa commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Mar 6, 2026

Uh oh!

cmsbuild commented Mar 6, 2026

Uh oh!

mmusich commented Mar 9, 2026

Uh oh!

mmusich Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

VourMa Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

mmusich Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

mmusich Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VourMa Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

VourMa Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

mmusich Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

VourMa Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented Mar 16, 2026

Uh oh!

cmsbuild commented Mar 16, 2026

Uh oh!

mmusich commented Mar 16, 2026

Uh oh!

cmsbuild commented Mar 18, 2026

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

Uh oh!

mmusich commented Mar 19, 2026

Uh oh!

mmusich commented Mar 19, 2026

Uh oh!

VourMa commented Mar 19, 2026

Uh oh!

civanch commented Mar 19, 2026

Uh oh!

gabrielmscampos commented Mar 19, 2026

Uh oh!

mmusich commented Mar 20, 2026

Uh oh!

AdrianoDee commented Mar 23, 2026

Uh oh!

cmsbuild commented Mar 23, 2026

Uh oh!

ftenchini commented Mar 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

VourMa commented Mar 6, 2026 •

edited

Loading

cmsbuild commented Mar 6, 2026 •

edited

Loading

mmusich Mar 15, 2026 •

edited

Loading