Run Scouting DQM in RelVals (+ change in JME scouting DQM sequence) by silviodonato · Pull Request #49358 · cms-sw/cmssw

silviodonato · 2025-11-10T16:38:38Z

PR description:

DQM Offline for Scouting was added by #47235.
It was activated in PromptReco at T0 only for ScoutingPFMonitor dataset by dmwm/T0#5053, specifically by adding @hltScouting to dqm_sequences in ProdOfflineConfiguration.py.
As discussed at the trigger review, it would be useful to have these DQM plots also in RelVals -- and also in AlCaVal -- to monitor the performance of HLT/Scouting objects.

Note: Currently Scouting DQM requires scouting object to run (otherwise it will crash). This PR changes the JME scouting DQM (jetMETDQMOfflineSourceScouting) to use Task. In this way the crashing module (ak4PFScoutingL1FastjetCorrector) will run only when requested by the JME scouting module (jetDQMAnalyzerSequenceScouting), ie. when scouting objects exists. In this way we avoid the crash (cc @etzovara).
Note-2: egmGsfElectronIDsForScoutingDQM will use gedGsfElectrons instead of slimmedElectrons to be compatible with RECO without AOD (cc @tihsu99)

This PR simply adds scouting DQM to the trigger DQM sequence.

PR validation:

I'm having some problems with the validation (#49331). I would like to check that all corner cases are covered (in particular I suspect that a re-reco of a non-scouting dataset might have problems)

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

To be defined

(previous attempts #49350 #49336)

FYI @patinkaew @cms-sw/hlt-l2

cmsbuild · 2025-11-10T16:39:04Z

cms-bot internal usage

cmsbuild · 2025-11-10T16:41:06Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49358/46748

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Tau DQM miniAOD validation for DATA #44029, Update BTV offline DQM sequences #46838

cmsbuild · 2025-11-10T16:41:27Z

A new Pull Request was created by @silviodonato for master.

It involves the following packages:

DQMOffline/Configuration (dqm)
DQMOffline/JetMET (dqm)
HLTriggerOffline/Scouting (dqm)

@cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@ahinzmann, @gkasieczka, @jdamgov, @jdolen, @mariadalfonso, @missirol, @mmarionncern, @mmusich, @mtosi, @nhanvtran, @rappoccio, @rociovilar, @schoef, @seemasharmafnal, @threus this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

silviodonato · 2025-11-10T16:44:33Z

please test

cmsbuild · 2025-11-10T18:41:19Z

-1

Failed Tests: RelVals AddOn
Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-255040/49379/summary.html
COMMIT: 239e63f
CMSSW: CMSSW_16_0_X_2025-11-10-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49358/49379/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

2024.0010001

----- Begin Fatal Exception 10-Nov-2025 18:13:54 CET-----------------------
An exception of category 'InvalidReference' occurred while
   [0] Processing  Event run: 379416 lumi: 1 event: 508034 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module ScoutingEGammaCollectionMonitoring/'ScoutingEGammaCollectionMonitoringOffline'
Exception Message:
ValueMap: no associated value for given product and index
----- End Fatal Exception -------------------------------------------------

2024.0070001

----- Begin Fatal Exception 10-Nov-2025 18:16:10 CET-----------------------
An exception of category 'InvalidReference' occurred while
   [0] Processing  Event run: 386478 lumi: 17 event: 22456638 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module ScoutingEGammaCollectionMonitoring/'ScoutingEGammaCollectionMonitoringOffline'
Exception Message:
ValueMap: no associated value for given product and index
----- End Fatal Exception -------------------------------------------------

2024.0050001

----- Begin Fatal Exception 10-Nov-2025 18:18:14 CET-----------------------
An exception of category 'InvalidReference' occurred while
   [0] Processing  Event run: 383811 lumi: 28 event: 14800950 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module ScoutingEGammaCollectionMonitoring/'ScoutingEGammaCollectionMonitoringOffline'
Exception Message:
ValueMap: no associated value for given product and index
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

Failed AddOn Tests

hlt_mc_Fake1

UNKNOWN

cmsbuild · 2025-11-10T19:12:09Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49358/46751

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Tau DQM miniAOD validation for DATA #44029, Update BTV offline DQM sequences #46838
- File HLTriggerOffline/Scouting/plugins/PatElectronTagProbeAnalyzer.cc modified in PR(s): [Do not merge] LLVM21 clang-tidy changes #49297

cmsbuild · 2025-11-10T19:12:27Z

Pull request #49358 was updated. @cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please check and sign again.

silviodonato · 2025-11-10T19:14:06Z

please test

cmsbuild · 2025-11-10T21:27:03Z

+1

Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-255040/49390/summary.html
COMMIT: c4165fe
CMSSW: CMSSW_16_0_X_2025-11-10-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49358/49390/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 10472 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 66 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3939953
DQMHistoTests: Total failures: 41
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3939892
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 41908.93 KiB( 50 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 2993.495 KiB HLT/ScoutingOffline
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-11-12T13:58:45Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-255040/49423/summary.html
COMMIT: d12c831
CMSSW: CMSSW_16_0_X_2025-11-12-1200/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49358/49423/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 33 errors in the following unit tests:

---> test TestDQMOfflineConfiguration_0 had ERRORS
---> test TestDQMOfflineConfiguration_110 had ERRORS
---> test TestDQMOfflineConfiguration_140 had ERRORS
and more ...

Failed RelVals

2023.00200012023.0020001_RunJetMET02023D_10k/step3_RunJetMET02023D_10k.log
2025.00000012025.0000001_RunZeroBias2025B_10k/step3_RunZeroBias2025B_10k.log
2024.00700012024.0070001_RunTau2024I_10k/step3_RunTau2024I_10k.log

Expand to see more relval errors ...

Failed RelVals-INPUT

138.5138.5_ExpressCollisions2021/step2_ExpressCollisions2021.log
134.0134.0_MinimumBiasBS/step2_MinimumBiasBS.log
2500.00022500.0002_NANOmc106Xul16APVv2/step2_NANOmc106Xul16APVv2.log

Expand to see more relval errors ...

silviodonato · 2025-11-12T14:31:20Z

It looks like I'm not able to run the Jet Correct "on demand" , when scouting objects are available.
I opened #49377 asking about this to the core team

cmsbuild · 2025-11-12T16:33:30Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49358/46795

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Tau DQM miniAOD validation for DATA #44029, Update BTV offline DQM sequences #46838

cmsbuild · 2025-11-12T16:33:56Z

Pull request #49358 was updated. @cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please check and sign again.

silviodonato · 2025-11-12T16:39:06Z

please test

cmsbuild · 2025-11-12T18:20:53Z

+1

Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-255040/49428/summary.html
COMMIT: 79c2446
CMSSW: CMSSW_16_0_X_2025-11-12-1200/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49358/49428/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 4 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 9 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3880468
DQMHistoTests: Total failures: 14
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3880434
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 708.916 KiB( 50 files compared)
DQMHistoSizes: changed ( 16834.0,... ): 177.229 KiB HLT/ScoutingOffline
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

silviodonato · 2025-11-12T18:35:03Z

The DQM of 2025.0010001_RunJetMET02025C contains the new scouting folder with non-empty plots.
The final workaround is to duplicate the scouting DQM sequence removing the JetCorrector which causes a crash when the scouting objects are missing.
I don't like this solution, but I think that for the time being it is better to have the scouting DQM without the JetCorrector than no scouting DQM at all.
Meanwhile I'm trying to find a long term solution in #49377.

mmusich · 2025-11-12T18:53:46Z

I wonder why though in the DQM bin-by-bin comparisons only the jet DQM shows: https://cern.ch/7qqq5.

cmsbuild · 2025-11-13T08:51:56Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49358/46804

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Tau DQM miniAOD validation for DATA #44029, Update BTV offline DQM sequences #46838

cmsbuild · 2025-11-13T08:52:20Z

Pull request #49358 was updated. @cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please check and sign again.

silviodonato · 2025-11-13T08:53:48Z

please test

(thanks Marco for spotting this bug!)

cmsbuild · 2025-11-13T10:53:27Z

+1

Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-255040/49432/summary.html
COMMIT: 330283e
CMSSW: CMSSW_16_0_X_2025-11-12-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49358/49432/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 900 lines to the logs
Reco comparison results: 10 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3880468
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3880441
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 11973.98 KiB( 50 files compared)
DQMHistoSizes: changed ( 16834.0,... ): 2993.495 KiB HLT/ScoutingOffline
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

silviodonato · 2025-11-13T11:06:34Z

I finally see the new plots at least for 2025_0010001.
It adds 904 lines to 2025.0000001_RunZeroBias2025B_10k because the scouting objects were unexpectedly missing (data from Emergency column with high randoms)
No crashes
I opened mayConsume runs prefetching #49377 to understand how to run the scouting jets corrector in DQM only when the scouting object is present

@mmusich @cms-sw/dqm-l2 do you have comments?

mmusich · 2025-11-13T11:13:24Z

It adds 904 lines to 2025.0000001_RunZeroBias2025B_10k because the scouting objects were unexpectedly missing (data from Emergency column with high randoms)

I guess we need PdmV to choose a better LS here (not sure if the data is available at CERN on disk).
I am not a huge fan of the duplicated sequence, but I guess it's OK for the time being, while we look for a better long term solution.
(incidentally this PR doesn't need hlt signature any longer, so if @cms-sw/dqm-l2 are OK I am also in favor of merging).

gabrielmscampos · 2025-11-14T08:44:23Z

+dqm

cmsbuild · 2025-11-14T08:44:47Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2025-11-14T15:17:35Z

+1

Martin-Grunewald · 2025-11-16T06:58:34Z

It seems this PR creates an error in the HLT-Validation test for HIon:

----- Begin Fatal Exception 15-Nov-2025 05:40:02 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 362321 lumi: 231 event: 111758081 stream: 1
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module VersionedGsfElectronIdProducer/'egmGsfElectronIDsForScoutingDQM'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: reco::GsfElectron
Looking for module label: slimmedElectrons
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

See here https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_16_0_X_2025-11-15-1100/el8_amd64_gcc13/RelVal_RECO_HIon_DATA.log

silviodonato · 2025-11-16T10:00:52Z

Hi @Martin-Grunewald ,
I'm trying to debug. How can reproduce the crash?

cd HLTrigger/Configuration/test
./cmsDriver.csh HIon
cmsRun RelVal_HLT_HIon_DATA.py

?
Which RelVal_Raw_HIon_DATA.root shall I use?

Martin-Grunewald · 2025-11-16T12:26:10Z

The IB tests: HLTrigger/Configuration/test/cmsDriver.csh ib

silviodonato · 2025-11-16T17:34:04Z

I added a fix in #49387 (comment)

Btw. in general we will have the crash in RECO,DQM steps without PAT in the middle, if scouting DQM is activated.
I noticed that in the matrix we always have RECO,PAT in standard pp collisions (see matrix), while we don't have PAT in cosmics and HIon

mmusich · 2025-12-15T10:43:08Z

I guess we need PdmV to choose a better LS here (not sure if the data is available at CERN on disk).

I opened #49627 to keep track of this.

AdrianoDee · 2026-03-23T10:12:20Z

Do I understand well that this

Note-2: egmGsfElectronIDsForScoutingDQM will use gedGsfElectrons instead of slimmedElectrons to be compatible with RECO without AOD (cc @tihsu99)

was not actually enforced? It seems the product label should be different. I'm asking since we are getting crashes in condition validation RelVals.

(Note that that RelVal includes PAT)

cmsbuild added this to the CMSSW_16_0_X milestone Nov 10, 2025

silviodonato mentioned this pull request Nov 10, 2025

Run Scouting DQM in RelVals (+ change in JME scouting DQM sequence) #49350

Closed

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending labels Nov 10, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 10, 2025

cmsbuild added tests-started and removed tests-pending labels Nov 10, 2025

cmsbuild added tests-rejected tests-pending code-checks-pending and removed tests-started tests-rejected code-checks-approved labels Nov 10, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 10, 2025

cmsbuild added tests-started and removed tests-pending labels Nov 10, 2025

fix in DQMOfflineScoutingForRelVals

79c2446

silviodonato mentioned this pull request Nov 12, 2025

mayConsume runs prefetching #49377

Open

use hltScoutingDqmOfflineForRelVals in DQMOfflineScoutingForRelVals

330283e

silviodonato mentioned this pull request Nov 16, 2025

Fix unit test crash in DQM/Integration [scouting_dqm_sourceclient-live_cfg.py] and HLT tests #49387

Merged

mmusich mentioned this pull request Dec 15, 2025

Use better input lumisection for wf 2025.0000001 #49627

Closed

etzovara mentioned this pull request Feb 25, 2026

JME Offline DQM: adding trigger efficiency for scouting jets #50188

Merged

etzovara mentioned this pull request Mar 24, 2026

[Backport] JME Offline DQM: adding trigger efficiency for scouting jets #50527

Merged

Conversation

silviodonato commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

cmsbuild commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Nov 10, 2025

Uh oh!

cmsbuild commented Nov 10, 2025

Uh oh!

silviodonato commented Nov 10, 2025

Uh oh!

cmsbuild commented Nov 10, 2025

Failed RelVals

Failed AddOn Tests

Uh oh!

cmsbuild commented Nov 10, 2025

Uh oh!

cmsbuild commented Nov 10, 2025

Uh oh!

silviodonato commented Nov 10, 2025

Uh oh!

cmsbuild commented Nov 10, 2025

Comparison Summary

Uh oh!

cmsbuild commented Nov 12, 2025

Failed Unit Tests

Failed RelVals

Failed RelVals-INPUT

Uh oh!

silviodonato commented Nov 12, 2025

Uh oh!

cmsbuild commented Nov 12, 2025

Uh oh!

cmsbuild commented Nov 12, 2025

Uh oh!

silviodonato commented Nov 12, 2025

Uh oh!

cmsbuild commented Nov 12, 2025

Comparison Summary

Uh oh!

silviodonato commented Nov 12, 2025

Uh oh!

mmusich commented Nov 12, 2025

Uh oh!

cmsbuild commented Nov 13, 2025

Uh oh!

cmsbuild commented Nov 13, 2025

Uh oh!

silviodonato commented Nov 13, 2025

Uh oh!

cmsbuild commented Nov 13, 2025

Comparison Summary

Uh oh!

silviodonato commented Nov 13, 2025

Uh oh!

mmusich commented Nov 13, 2025

Uh oh!

gabrielmscampos commented Nov 14, 2025

Uh oh!

cmsbuild commented Nov 14, 2025

Uh oh!

mandrenguyen commented Nov 14, 2025

Uh oh!

Martin-Grunewald commented Nov 16, 2025

Uh oh!

silviodonato commented Nov 16, 2025

Uh oh!

Martin-Grunewald commented Nov 16, 2025

Uh oh!

silviodonato commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented Dec 15, 2025

Uh oh!

AdrianoDee commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

silviodonato commented Nov 10, 2025 •

edited

Loading

cmsbuild commented Nov 10, 2025 •

edited

Loading

silviodonato commented Nov 16, 2025 •

edited

Loading

AdrianoDee commented Mar 23, 2026 •

edited

Loading