update phase-2 HLT timing script and use `alpaka` modifier in NGT workflows by mmusich · Pull Request #49821 · cms-sw/cmssw

mmusich · 2026-01-14T13:03:07Z

PR description:

Following the discussion had at the NGT meeting of Jan 13 2025 this PR implements:

use explicitly the alpaka process modifier in all the NGT scouting related workflows offload part of the HGCal reconstruction to GPUs

PR validation:

To be tested by the bot.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, no backport needed.

mmusich · 2026-01-14T13:03:16Z

type ngt

cmsbuild · 2026-01-14T13:03:35Z

cms-bot internal usage

cmsbuild · 2026-01-14T13:04:59Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49821/47477

cmsbuild · 2026-01-14T13:05:23Z

A new Pull Request was created by @mmusich for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv)
HLTrigger/Configuration (hlt)

@AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @cmsbuild, @miquork, @mmusich can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @SohamBhattacharya, @VourMa, @fabiocos, @makortel, @missirol, @rovere, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2026-01-14T13:05:48Z

test parameters:

enable = hlt_p2_integration, hlt_p2_timing
workflows = ph2_hlt

mmusich · 2026-01-14T13:06:00Z

@cmsbuild, please test

cmsbuild · 2026-01-14T14:20:43Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-480d8a/50620/summary.html
COMMIT: 43410e9
CMSSW: CMSSW_16_1_X_2026-01-13-2300/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49821/50620/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Failed RelVals

34634.999

----- Begin Fatal Exception 14-Jan-2026 14:46:42 CET-----------------------
An exception of category 'OutOfBound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 0
   [1] Running path 'HLTriggerFinalPath'
   [2] Prefetching for module TriggerSummaryProducerAOD/'hltTriggerSummaryAOD'
   [3] Prefetching for module L1HPSPFTauProducer/'l1tHPSPFTauProducer'
   [4] Prefetching for module L1TPFCandMultiMerger/'l1tLayer1'
   [5] Prefetching for module L1TCorrelatorLayer1Producer/'l1tLayer1HGCal'
   [6] Calling method for module HGCalBackendLayer2Producer/'l1tHGCalBackEndLayer2Producer'
Exception Message:
TC X1 = 0.0713466 out of the seeding histogram bounds 0.076 - 0.58
----- End Fatal Exception -------------------------------------------------

mmusich · 2026-01-15T09:00:41Z

The updated timing result, adding the alpaka process modifier is odd:

mmusich · 2026-01-15T10:59:37Z

@cmsbuild, please test

cmsbuild · 2026-01-15T14:24:13Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-480d8a/50652/summary.html
COMMIT: 43410e9
CMSSW: CMSSW_16_1_X_2026-01-14-2300/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49821/50652/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially added 5 lines to the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 71
DQMHistoTests: Total histograms compared: 4580671
DQMHistoTests: Total failures: 1010
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4579641
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 70 files compared)
Checked 285 log files, 240 edm output root files, 71 DQM output files
TriggerResults: found differences in 3 / 69 workflows

cmsbuild · 2026-03-02T18:38:40Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49821/48331

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): TICL: Consolidate v5 as Default Configuration and Cleanup Legacy Code #49932, Switch to the new tracking baseline (single iteration, CA-extended Patatrack + LST, mkFit) as Phase 2 HLT default #50040, CloseBy smearing for Phase-1. #50225
- File HLTrigger/Configuration/python/HLT_75e33/test/runHLTTiming.sh modified in PR(s): Switch to the new tracking baseline (single iteration, CA-extended Patatrack + LST, mkFit) as Phase 2 HLT default #50040

cmsbuild · 2026-03-02T18:39:06Z

Pull request #49821 was updated. @AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @cmsbuild, @miquork, @mmusich can you please check and sign again.

mmusich · 2026-03-02T18:39:08Z

@cmsbuild, please test

let's try again to test this now that Update Phase 2 timing configuration #50278 is merged

cmsbuild · 2026-03-02T20:51:50Z

-1

Failed Tests: HLTP2Timing
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-480d8a/51704/summary.html
COMMIT: 60e4b39
CMSSW: CMSSW_16_1_X_2026-03-02-1100/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49821/51704/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 5 lines to the logs
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 72
DQMHistoTests: Total histograms compared: 4736055
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4736035
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 71 files compared)
Checked 292 log files, 246 edm output root files, 72 DQM output files
TriggerResults: found differences in 3 / 70 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 2 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34434.77_TTbar_14TeV+Run4D121_NGTScouting step2 max memory diff 152.3 exceeds +/- 90.0 MiB
Error: Workflow 34434.772_TTbar_14TeV+Run4D121_NGTScoutingWithNano step2 max memory diff 153.1 exceeds +/- 90.0 MiB

cmsbuild · 2026-04-04T12:41:02Z

Milestone for this pull request has been moved to CMSSW_17_0_X. Please open a backport if it should also go in to CMSSW_16_1_X.

cmsbuild added this to the CMSSW_16_1_X milestone Jan 14, 2026

cmsbuild added hlt-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending ngt labels Jan 14, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 14, 2026

cmsbuild added tests-started and removed tests-pending labels Jan 14, 2026

cmsbuild mentioned this pull request Jan 14, 2026

Remove alpaka procModfier from workflows in which it is no longer useful #49755

Closed

cmsbuild added tests-rejected and removed tests-started labels Jan 14, 2026

This was referenced Jan 14, 2026

Deprecate autoCondPhase2 keys for inexistent geometries #49790

Merged

CPU vs. GPU for LST in HLT and updates to the offline #49832

Closed

cmsbuild added tests-started and removed tests-rejected labels Jan 15, 2026

cmsbuild removed the tests-started label Jan 15, 2026

cmsbuild added the code-checks-pending label Mar 2, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 2, 2026

cmsbuild added tests-started and removed tests-pending labels Mar 2, 2026

cmsbuild added tests-rejected and removed tests-started labels Mar 2, 2026

This was referenced Mar 3, 2026

Particle Flow cluster validation in ECAL #50292

Open

MLPF Run 3 inference code and model update #50316

Merged

TRK CPU vs. GPU path and infrastructure for Phase 2 HLT #50336

Merged

This was referenced Mar 13, 2026

update runHLTTiming script to be more modular #50398

Merged

Introduce Phase 2 FastSim (tracker) workflows #50437

Merged

add GPU memory and usage to the phase2 HLT timing check #50479

Merged

cmsbuild mentioned this pull request Mar 27, 2026

Rpc digi dev v9 #47447

Open

cmsbuild modified the milestones: CMSSW_16_1_X, CMSSW_17_0_X Apr 4, 2026

cmsbuild mentioned this pull request Apr 10, 2026

TICL-barrel validation @ HLT #50588

Merged

cmsbuild mentioned this pull request Apr 17, 2026

Phase2 HLT nano workflows: miscelleaneous fixes for TICL-related modules. #50748

Merged

This was referenced May 4, 2026

hlt-p2-timing: Drop live output from runHLTTiming.sh #50867

Closed

Fix hlt-p2-timing: properly wait and comsume the background processed #50868

Merged

Sync FASTPU_X changes in to main CMSSW #50886

Open

Conversation

mmusich commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

mmusich commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

mmusich commented Jan 14, 2026

Uh oh!

mmusich commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Failed RelVals

Uh oh!

mmusich commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented Jan 15, 2026

Uh oh!

cmsbuild commented Jan 15, 2026

Comparison Summary

Uh oh!

cmsbuild commented Mar 2, 2026

Uh oh!

cmsbuild commented Mar 2, 2026

Uh oh!

mmusich commented Mar 2, 2026

Uh oh!

cmsbuild commented Mar 2, 2026

Comparison Summary

Max Memory Comparisons exceeding threshold

Uh oh!

cmsbuild commented Apr 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mmusich commented Jan 14, 2026 •

edited

Loading

cmsbuild commented Jan 14, 2026 •

edited

Loading

mmusich commented Jan 15, 2026 •

edited

Loading