[NGT] Add utility to produce plots from DQM output files

[Parsifal-2045]

PR description:

This PR introduces a new utility script, dqm-plot, located in DQMServices/Components/scripts/. This tool allows users to easily compare DQM histograms from multiple ROOT files using mplhep with official CMS plotting styles.

Main features:

• Supports regex patterns to select and filter specific histograms (or groups of histograms) from input files
• Allows to overlay plots from different folders for easier comparison
• Uses mplhep and standard CMS style guidelines (CMS logo, energy labels, font styles, color palette)
• Automatically generates ratio plots, handles normalization and log scales as well as legend placing (implementation is as generic as possible, there might be cases where fonts or placing are not optimal)
• Implements multiprocessing to parallelize the plotting of large numbers of histograms.
• Optional generation of index.php files for easy deployment and viewing on web servers (e.g., CERN web space).
• Supports output in both PNG and PDF formats.

Usage Example:

dqm-plot \
--web \
--pdf \
-s "DQMData/Run 1/HLT/Run summary/Muon/*" \
--energy-text "ZMM 200PU (14 TeV)" \
--overlay "hltPhase2L3FromL1TkMuonPixelTrks:hltIter2Phase2L3FromL1TkMuonMerged" \
--legend "File 1, File 2" \
file1.root file2.root

There are more available options, all accessible in the help message of the utility

dqm-plot --help

An example of the output from Phase 2 HLT Validation DQM files can be found here

PR validation:

The script has been tested locally with standard DQM ROOT files

[cmsbuild]

cms-bot internal usage

[Parsifal-2045]

type ngt

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49830/47483

[cmsbuild]

A new Pull Request was created by @Parsifal-2045 for master.

It involves the following packages:

• DQMServices/Components (dqm)

@cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[nothingface0]

@Parsifal-2045 Thanks for this! How crucial do you think this tool is/will be? Perhaps it's a stretch, but could you add a test to make sure that the basic script functionality stays correct in the future? On the other hand, I don't know if it's a good idea to break the whole CMSSW tests in case this test breaks.

[Parsifal-2045]

@Parsifal-2045 Thanks for this! How crucial do you think this tool is/will be? Perhaps it's a stretch, but could you add a test to make sure that the basic script functionality stays correct in the future? On the other hand, I don't know if it's a good idea to break the whole CMSSW tests in case this test breaks.

I don't think I can really comment on how crucial it will be. I have been using it extensively for the past few months and it has proven very useful for checking physics performance during development. I have also shared it with some colleagues and many of the features here have been added thanks to their feedback (as well as my own desiderata).
I was also not sure about the testing since it's technically just a plotting tool, breaking CMSSW for e.g. a change in Python version seems a bit too aggressive. However, I'd be happy to discuss some possible solutions or middle-grounds

[mmusich]

I was also not sure about the testing since it's technically just a plotting tool, breaking CMSSW for e.g. a change in Python version seems a bit too aggressive.

i think something along the lines of #49817 is a very reasonable request (here you would need two input files).

[Parsifal-2045]

I was also not sure about the testing since it's technically just a plotting tool, breaking CMSSW for e.g. a change in Python version seems a bit too aggressive.

i think something along the lines of #49817 is a very reasonable request (here you would need two input files).

I have just added a unit test along the lines of the suggestion (thank you!). I ended up using a single file twice, but with overlaid plots to also test ratios that are not 1

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49830/47490

[cmsbuild]

Pull request #49830 was updated. @cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please check and sign again.

[mmusich]

Assuming no hurry about integration, I suggest to test with tomorrow's IB as the development branch is still turbolent

[mmusich]

@cmsbuild, please test

[cmsbuild]

+1

Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7c9f9/50645/summary.html
COMMIT: be68421
CMSSW: CMSSW_16_1_X_2026-01-14-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49830/50645/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 2 lines to the logs
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 52
• DQMHistoTests: Total histograms compared: 4025536
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4025513
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
• Checked 222 log files, 193 edm output root files, 52 DQM output files
• TriggerResults: no differences found

[nothingface0]

+dqm
Thanks @Parsifal-2045 !

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1