Update BTV offline DQM sequences by mmarchegiani · Pull Request #46838 · cms-sw/cmssw

mmarchegiani · 2024-12-02T09:28:41Z

PR description:

This PR updates the DQM sequences corresponding to the offline BTV DQM. Following the recent PR #46674 to include the DQM histograms of the newest Run-3 b-taggers, the DQMOfflineBTag sequence is redefined such that the bTagMiniDQMSource producing the DQM histograms for the Run-3 taggers is executed. The harvesting step of the DQM is modified accordingly such that the DQMHarvestBTag sequence is executed to fill the histograms.
These changes are needed such that the alias btag defined in DQMOffline/Configuration/python/autoDQM.py, including the sequences DQMOfflineBTag and DQMHarvestBTag, is running the production of the offline BTV DQM histograms.
After this PR is merged, the tag @btag can be used in https://github.com/dmwm/T0/blob/master/etc/ProdOfflineConfiguration.py to run the BTV offline DQM on datasets at Tier-0.

PR validation:

In order to test that the @btag tag is running the updated DQM sequences, the following command on the Run2024D/BTagMu dataset is run to run the prompt reconstruction and DQM steps:

python3 $CMSSW_RELEASE_BASE/src/Configuration/DataProcessing/test/RunPromptReco.py --scenario=ppEra_Run3 --reco --aod --miniaod --global-tag 140X_dataRun3_Prompt_v2 --lfn=file:/eos/user/m/mmarcheg/BTVOfflineDQM/store/data/Run2024D/BTagMu/RAW/v1/000/380/649/00000/ff51f62b-a054-4b81-a6f3-58996b9044f6.root --alcarecos=TkAlMinBias+TkAlV0s --PhysicsSkims=LogError+LogErrorMonitor --dqmio --dqmSeq=@common+@btag

Finally, the harvesting step is run with the following command:

cmsDriver.py step4  -s HARVESTING:@btag --conditions auto:run3_data --data  --filetype DQM --scenario pp --era Run3_2024 -n 100 --filein file:output_inDQMIO.root --fileout file:step4.root

The output file DQM_V0001_R000380649__Global__CMSSW_X_Y_Z__RECO.root is then opened with ROOT and explored with TBrowser to ensure that the BTV DQM histograms are correctly saved in the output.

The generic tests described in https://cms-sw.github.io/PRWorkflow.html have been executed. All the tests run with scram b runtests use-ibeos are successful. Most of the tests run with runTheMatrix.py -l limited -i all --ibeos are successful, with some tests failed due to Step0-DAS_ERROR:

2022.000001_RunJetHT2022C_10k Step0-DAS_ERROR Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN  - time date Thu Nov 28 19:38:13 2024-date Thu Nov 28 19:38:10 2024; exit: 256 0 0 0
2022.002001_RunZeroBias2022C_10k Step0-DAS_ERROR Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN  - time date Thu Nov 28 19:38:16 2024-date Thu Nov 28 19:38:14 2024; exit: 256 0 0 0
2023.002001_RunZeroBias2023D_10k Step0-DAS_ERROR Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN  - time date Thu Nov 28 19:38:18 2024-date Thu Nov 28 19:38:17 2024; exit: 256 0 0 0
2024.000001_RunZeroBias2024B_10k Step0-DAS_ERROR Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN  - time date Thu Nov 28 19:38:21 2024-date Thu Nov 28 19:38:19 2024; exit: 256 0 0 0
2024.101001_RunBTagMu2024C_10k Step0-DAS_ERROR Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN  - time date Thu Nov 28 19:38:23 2024-date Thu Nov 28 19:38:22 2024; exit: 256 0 0 0
2024.202001_RunJetMET02024D_10k Step0-DAS_ERROR Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN  - time date Thu Nov 28 19:38:26 2024-date Thu Nov 28 19:38:24 2024; exit: 256 0 0 0
2024.303001_RunDisplacedJet2024E_10k Step0-DAS_ERROR Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN  - time date Thu Nov 28 19:38:27 2024-date Thu Nov 28 19:38:26 2024; exit: 256 0 0 0

However, the Step0-DAS_ERROR is related to fetching the datasets and is not an error specific to our changes, therefore we open the PR.

cmsbuild · 2024-12-02T09:29:06Z

cms-bot internal usage

cmsbuild · 2024-12-02T09:31:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46838/42858

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_SecondStep_cff.py modified in PR(s): Tau DQM miniAOD validation for DATA #44029
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Tau DQM miniAOD validation for DATA #44029

cmsbuild · 2024-12-02T09:31:21Z

A new Pull Request was created by @mmarchegiani for master.

It involves the following packages:

DQMOffline/Configuration (dqm)

@antoniovagnerini, @cmsbuild, @rseidita can you please review it and eventually sign? Thanks.
@rociovilar, @threus this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

rseidita · 2024-12-02T09:33:50Z

please test

cmsbuild · 2024-12-02T10:36:22Z

-1

Failed Tests: RelVals RelVals-INPUT
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d376c/43183/summary.html
COMMIT: 7ab6ab4
CMSSW: CMSSW_15_0_X_2024-12-01-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46838/43183/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

140.56

----- Begin Fatal Exception 02-Dec-2024 10:57:28 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1394020 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module MiniAODTaggerAnalyzer/'bTagDeepFlavourDQMBvsAllGlobalAnalyzer'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: std::vector<pat::Jet>
Looking for module label: slimmedJetsPuppi
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

138.4138.4_PromptCollisions2021/step2_PromptCollisions2021.log
140.61140.61_RunHI2022/step2_RunHI2022.log
142.903142.903_RunUPC2016PA/step2_RunUPC2016PA.log

Expand to see more relval errors ...

antoniovagnerini · 2024-12-06T09:46:52Z

please test

cmsbuild · 2024-12-06T10:47:10Z

-1

Failed Tests: RelVals RelVals-INPUT
Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d376c/43294/summary.html
COMMIT: 7ab6ab4
CMSSW: CMSSW_15_0_X_2024-12-05-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46838/43294/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

140.56

----- Begin Fatal Exception 06-Dec-2024 11:02:44 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1394020 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module MiniAODTaggerAnalyzer/'bTagDeepFlavourDQMBvsAllGlobalAnalyzer'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: std::vector<pat::Jet>
Looking for module label: slimmedJetsPuppi
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

138.4138.4_PromptCollisions2021/step2_PromptCollisions2021.log
140.61140.61_RunHI2022/step2_RunHI2022.log
142.903142.903_RunUPC2016PA/step2_RunUPC2016PA.log

Expand to see more relval errors ...

cmsbuild · 2025-02-07T08:32:51Z

Milestone for this pull request has been moved to CMSSW_15_1_X. Please open a backport if it should also go in to CMSSW_15_0_X.

nothingface0 · 2025-09-26T13:31:36Z

@mmarchegiani what is the status of this PR? Are you planning to keep working on it? It looks like some of the workflows are failing.

Dimitris for DQM

cmsbuild · 2025-12-18T12:58:56Z

Milestone for this pull request has been moved to CMSSW_16_1_X. Please open a backport if it should also go in to CMSSW_16_0_X.

cmsbuild · 2026-04-04T12:39:58Z

Milestone for this pull request has been moved to CMSSW_17_0_X. Please open a backport if it should also go in to CMSSW_16_1_X.

cmsbuild · 2026-04-04T12:41:42Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d376c/52472/summary.html
COMMIT: 7ab6ab4
CMSSW: CMSSW_16_1_X_2026-04-04-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46838/52472/install.sh to create a dev area with all the needed externals and cmssw changes.

This pull request cannot be automatically merged, could you please rebase it?
You can see the log for git cms-merge-topic here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d376c/52472/git-merge-result

mmarchegiani added 3 commits November 21, 2024 16:40

Redefine DQMOfflineBTag to include BTag DQM sequence with Run-3 taggers

e834084

Redefine DQMHarvestBTag to include BTag DQM sequence with Run-3 taggers

d4626e3

Replace old harvest sequence with new one

7ab6ab4

cmsbuild added this to the CMSSW_15_0_X milestone Dec 2, 2024

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending labels Dec 2, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 2, 2024

cmsbuild added tests-started and removed tests-pending labels Dec 2, 2024

cmsbuild added tests-rejected and removed tests-started labels Dec 2, 2024

cmsbuild added tests-started and removed tests-rejected labels Dec 6, 2024

cmsbuild added tests-rejected and removed tests-started labels Dec 6, 2024

This was referenced Jan 31, 2025

Scouting Offline DQM framework for muons #47207

Closed

Scouting Offline DQM for 2025 data-taking (EGM+MUO+autoDQM) #47235

Merged

cmsbuild removed this from the CMSSW_15_0_X milestone Feb 7, 2025

cmsbuild mentioned this pull request Sep 24, 2025

add HLT vertexing resolution monitoring (by split vertex method) #48980

Merged

cmsbuild mentioned this pull request Oct 8, 2025

Implement GPU vs CPU comparison for HLT heterogeneous products in patatrack workflows #49105

Merged

This was referenced Nov 16, 2025

Fix unit test crash in DQM/Integration [scouting_dqm_sourceclient-live_cfg.py] and HLT tests #49387

Merged

ECAL Phase 2 - Upgrades for Phase 2 development workflow, validation and DQM #49409

Merged

cmsbuild mentioned this pull request Nov 24, 2025

[NGT] Introduce DQM for HLT nano #49450

Merged

cmsbuild modified the milestones: CMSSW_16_0_X, CMSSW_16_1_X Dec 18, 2025

cmsbuild mentioned this pull request Feb 27, 2026

PF DQM updates #50282

Open

cmsbuild modified the milestones: CMSSW_16_1_X, CMSSW_17_0_X Apr 4, 2026

cmsbuild added tests-started and removed tests-rejected labels Apr 4, 2026

cmsbuild added tests-rejected and removed tests-started labels Apr 4, 2026

cmsbuild modified the milestones: CMSSW_16_1_X, CMSSW_17_0_X Apr 4, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update BTV offline DQM sequences#46838

Update BTV offline DQM sequences#46838
mmarchegiani wants to merge 3 commits intocms-sw:masterfrom
mmarchegiani:from-CMSSW_14_2_0_pre4

mmarchegiani commented Dec 2, 2024

Uh oh!

cmsbuild commented Dec 2, 2024 •

edited

Loading

Uh oh!

cmsbuild commented Dec 2, 2024

Uh oh!

cmsbuild commented Dec 2, 2024

Uh oh!

rseidita commented Dec 2, 2024

Uh oh!

cmsbuild commented Dec 2, 2024

Uh oh!

antoniovagnerini commented Dec 6, 2024

Uh oh!

cmsbuild commented Dec 6, 2024

Uh oh!

cmsbuild commented Feb 7, 2025

Uh oh!

nothingface0 commented Sep 26, 2025

Uh oh!

cmsbuild commented Dec 18, 2025

Uh oh!

cmsbuild commented Apr 4, 2026

Uh oh!

cmsbuild commented Apr 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mmarchegiani commented Dec 2, 2024

PR description:

PR validation:

Uh oh!

cmsbuild commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Dec 2, 2024

Uh oh!

cmsbuild commented Dec 2, 2024

Uh oh!

rseidita commented Dec 2, 2024

Uh oh!

cmsbuild commented Dec 2, 2024

RelVals

RelVals-INPUT

Uh oh!

antoniovagnerini commented Dec 6, 2024

Uh oh!

cmsbuild commented Dec 6, 2024

RelVals

RelVals-INPUT

Uh oh!

cmsbuild commented Feb 7, 2025

Uh oh!

nothingface0 commented Sep 26, 2025

Uh oh!

cmsbuild commented Dec 18, 2025

Uh oh!

cmsbuild commented Apr 4, 2026

Uh oh!

cmsbuild commented Apr 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cmsbuild commented Dec 2, 2024 •

edited

Loading