Updates to the offline CPU vs. GPU workflows for LST by VourMa · Pull Request #49984 · cms-sw/cmssw

VourMa · 2026-01-29T22:11:18Z

This PR applies the additional CPU reconstruction (SerialSync) and CPU vs. GPU comparison plots for the LST implementation in the Phase 2 offline reconstruction through a new procModifier, alpakaValidationLST. The new modifier is also included in the alpakaValidation modifier chain.

With the introduction of the alpakaValidationLST modifier, the offline workflow testing LST on CPU vs. LST on GPU can be made explicit. The code is changed so that the heterogeneous workflow 0.712 (previously 0.704) runs the offline reconstruction without any additional CPU reconstruction, while a new workflow, 0.713, runs the comparison. Workflow 0.703 has also been renamed to 0.711. The workflow numbering changes are made so that the offline LST workflows follow the numbering conventions for Alpaka workflows, following #49832 (comment).

The PR has been validated by running workflows 0.711, 0.712 and 0.713 locally. They finished successfully, with the anticipated content in the DQM file.

This PR includes the offline part of #49832 and it supersedes it. It requires and is required by cms-sw/cms-bot#2663.

cmsbuild · 2026-01-29T22:11:46Z

cms-bot internal usage

cmsbuild · 2026-01-29T22:13:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49984/47771

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/ProcessModifiers/python/alpakaValidationLST_cff.py modified in PR(s): CPU vs. GPU for LST in HLT and updates to the offline #49832
- File Configuration/ProcessModifiers/python/alpakaValidation_cff.py modified in PR(s): CPU vs. GPU for LST in HLT and updates to the offline #49832
- File Configuration/PyReleaseValidation/README.md modified in PR(s): Remove alpaka procModfier from workflows in which it is no longer useful #49755, CPU vs. GPU for LST in HLT and updates to the offline #49832, TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File Configuration/PyReleaseValidation/python/relval_Run4.py modified in PR(s): Remove alpaka procModfier from workflows in which it is no longer useful #49755, CPU vs. GPU for LST in HLT and updates to the offline #49832, TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File Configuration/PyReleaseValidation/python/relval_gpu.py modified in PR(s): CPU vs. GPU for LST in HLT and updates to the offline #49832
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Remove alpaka procModfier from workflows in which it is no longer useful #49755, update phase-2 HLT timing script and use alpaka modifier in NGT workflows #49821, CPU vs. GPU for LST in HLT and updates to the offline #49832, TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932
- File DQM/TrackingMonitorClient/python/TrackingClientConfig_Tier0_cff.py modified in PR(s): CPU vs. GPU for LST in HLT and updates to the offline #49832
- File DQM/TrackingMonitorSource/python/trackToTrackCPU_cff.py modified in PR(s): CPU vs. GPU for LST in HLT and updates to the offline #49832
- File RecoTracker/IterativeTracking/python/HighPtTripletStep_cff.py modified in PR(s): CPU vs. GPU for LST in HLT and updates to the offline #49832
- File RecoTracker/LSTCore/standalone/README.md modified in PR(s): CPU vs. GPU for LST in HLT and updates to the offline #49832

cmsbuild · 2026-01-29T22:13:59Z

A new Pull Request was created by @VourMa for master.

It involves the following packages:

Configuration/ProcessModifiers (operations)
Configuration/PyReleaseValidation (pdmv)
DQM/TrackingMonitorClient (dqm)
DQM/TrackingMonitorSource (dqm)
RecoTracker/IterativeTracking (reconstruction)
RecoTracker/LSTCore (reconstruction)

@AdrianoDee, @DickyChant, @Moanwar, @antoniovagnerini, @cmsbuild, @ctarricone, @davidlange6, @fabiocos, @ftenchini, @gabrielmscampos, @jfernan2, @mandrenguyen, @miquork, @nothingface0, @rseidita, @srimanob can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @Martin-Grunewald, @VinInn, @VourMa, @arossi83, @dgulhan, @elusian, @fabiocos, @felicepantaleo, @fioriNTU, @gpetruc, @idebruyn, @jandrea, @makortel, @mmasciov, @mmusich, @mtosi, @richa2710, @rovere, @slomeo, @sroychow, @threus this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

nothingface0 · 2026-01-30T08:28:00Z

@cmsbuild please test

VourMa · 2026-01-30T09:29:47Z

@nothingface0 I think we would need something the below to properly test this (I don't have bot permissions but maybe you can make it work):

enable = gpu
pull_request = Update GPU wfs for 1601 after renumbering in cmssw cms-bot#2663
workflows_gpu = 34434.712, 34434.713
relvals_opt = -w upgrade,standard
relvals_opt_gpu = -w upgrade,standard

nothingface0 · 2026-01-30T10:08:37Z

@cmsbuild abort

nothingface0 · 2026-01-30T10:08:59Z

test parameters:

pull_request = Update GPU wfs for 1601 after renumbering in cmssw cms-bot#2663
workflows_gpu = 34434.712, 34434.713
relvals_opt = -w upgrade,standard
relvals_opt_gpu = -w upgrade,standard

nothingface0 · 2026-01-30T10:10:45Z

enable gpu

nothingface0 · 2026-01-30T10:11:30Z

@cmsbuild please test

@VourMa sorry, I got ahead of myself 😅

AdrianoDee · 2026-02-05T08:28:22Z

+pdmv

AdrianoDee · 2026-02-05T08:29:23Z

test parameters:

pull_request = Update GPU wfs for 1601 after renumbering in cmssw cms-bot#2663, Patch to fix workflow errors from HGCalTypes changes #50039
relvals_opt = -w upgrade,standard
relvals_opt_gpu = -w upgrade,standard

AdrianoDee · 2026-02-05T08:30:21Z

please test
(sorry to use this as a guinea pig, but I think it's worth testing if the bug fix works properly here)

cmsbuild · 2026-02-05T12:37:48Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-43406e/51117/summary.html
COMMIT: fefacea
CMSSW: CMSSW_16_1_X_2026-02-04-2300/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49984/51117/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 6760 differences found in the comparisons
DQMHistoTests: Total files compared: 54
DQMHistoTests: Total histograms compared: 4263089
DQMHistoTests: Total failures: 18427
DQMHistoTests: Total nulls: 28
DQMHistoTests: Total successes: 4244614
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 53 files compared)
Checked 230 log files, 203 edm output root files, 54 DQM output files
TriggerResults: found differences in 3 / 52 workflows

AMD_MI300X Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

AMD_W7900 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

gabrielmscampos · 2026-02-05T12:53:11Z

+dqm

Moanwar · 2026-02-05T13:35:41Z

+1

cmsbuild · 2026-02-05T13:36:10Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cms-bot#2663, #50039

ftenchini · 2026-02-09T21:39:52Z

+1

Updates to the offline CPU vs. GPU workflows for LST

fefacea

cmsbuild added this to the CMSSW_16_1_X milestone Jan 29, 2026

cmsbuild added reconstruction-pending dqm-pending operations-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending tracking labels Jan 29, 2026

VourMa mentioned this pull request Jan 29, 2026

CPU vs. GPU for LST in HLT and updates to the offline #49832

Closed

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 29, 2026

VourMa mentioned this pull request Jan 29, 2026

Update GPU wfs for 1601 after renumbering in cmssw cms-sw/cms-bot#2663

Merged

cmsbuild added tests-started and removed tests-pending labels Jan 30, 2026

cmsbuild added tests-pending requires-external and removed tests-started labels Jan 30, 2026

cmsbuild removed the tests-pending label Jan 30, 2026

cmsbuild added pdmv-approved and removed pdmv-pending labels Feb 5, 2026

cmsbuild added operations-pending tests-started and removed operations-approved tests-approved tests-ib-failure labels Feb 5, 2026

cmsbuild mentioned this pull request Feb 5, 2026

Switch to the new tracking baseline (single iteration, CA-extended Patatrack + LST, mkFit) as Phase 2 HLT default #50040

Merged

cmsbuild added operations-approved tests-approved and removed operations-pending tests-started labels Feb 5, 2026

AdrianoDee mentioned this pull request Feb 5, 2026

Patch to fix workflow errors from HGCalTypes changes #50039

Closed

cmsbuild added dqm-approved and removed dqm-pending labels Feb 5, 2026

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels Feb 5, 2026

cmsbuild added orp-approved and removed orp-pending labels Feb 9, 2026

cmsbuild merged commit ea41fb2 into cms-sw:master Feb 9, 2026
23 checks passed

Conversation

VourMa commented Jan 29, 2026

Uh oh!

cmsbuild commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 29, 2026

Uh oh!

cmsbuild commented Jan 29, 2026

Uh oh!

nothingface0 commented Jan 30, 2026

Uh oh!

VourMa commented Jan 30, 2026

Uh oh!

nothingface0 commented Jan 30, 2026

Uh oh!

nothingface0 commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nothingface0 commented Jan 30, 2026

Uh oh!

nothingface0 commented Jan 30, 2026

Uh oh!

AdrianoDee commented Feb 5, 2026

Uh oh!

AdrianoDee commented Feb 5, 2026

Uh oh!

AdrianoDee commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Feb 5, 2026

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

Uh oh!

gabrielmscampos commented Feb 5, 2026

Uh oh!

Moanwar commented Feb 5, 2026

Uh oh!

cmsbuild commented Feb 5, 2026

Uh oh!

ftenchini commented Feb 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cmsbuild commented Jan 29, 2026 •

edited

Loading

nothingface0 commented Jan 30, 2026 •

edited

Loading

AdrianoDee commented Feb 5, 2026 •

edited

Loading