Skip to content

2026 RelVal Workflows#49419

Merged
cmsbuild merged 2 commits into
cms-sw:masterfrom
AdrianoDee:relval_2026_wfs
Nov 24, 2025
Merged

2026 RelVal Workflows#49419
cmsbuild merged 2 commits into
cms-sw:masterfrom
AdrianoDee:relval_2026_wfs

Conversation

@AdrianoDee
Copy link
Copy Markdown
Contributor

PR description:

The 2026 version of #46445.

This PR proposes the addition of 2026 wfs to the matrix. For the moment, they are a copy of what we run currently for 2025 with the idea that once 2026 parameters (conditions, HLT menu) come, we will need to fill the dummy ones (keeping them in sync with each other). For the moment:

  • @relval2026 is pointing toGRun (@cms-sw/hlt-l2, let me know if you prefer something different there);
  • auto:phase1_2026_* symbolic GTs are copies of 2025 ones (@cms-sw/alca-l2 let me know what you think about this, I'll updated them following your suggestion);

The era is Era_2026 that, for now, is just a copy of Era_2025 (the same for FastSim, just fyi @cms-sw/simulation-l2). TTbar 2026 noPU/PU wfs are added to the limited matrix to be tested with PRs. The standard PU input for 2026 wfs is the same as 2025 for now. I took the chance to update also the GPU workflows to 2026 (@cms-sw/heterogeneous-l2) so that we avoid running them with the fake menu when 2025 wfs will be switching to that (as happened last year).

PR validation:

New wfs run.

@AdrianoDee
Copy link
Copy Markdown
Contributor Author

enable gpu

@cmsbuild
Copy link
Copy Markdown
Contributor

cmsbuild commented Nov 19, 2025

cms-bot internal usage

@AdrianoDee
Copy link
Copy Markdown
Contributor Author

please test
(the new tests are in the limited matrix now)

@cmsbuild
Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49419/46876

@cmsbuild
Copy link
Copy Markdown
Contributor

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

  • Configuration/AlCa (alca)
  • Configuration/Eras (operations)
  • Configuration/HLT (hlt)
  • Configuration/PyReleaseValidation (pdmv)

@AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @arunhep, @atpathak, @cmsbuild, @davidlange6, @fabiocos, @ftenchini, @mandrenguyen, @miquork, @mmusich, @perrotta can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @mmusich, @rsreds, @slomeo, @tocheng, @yuanchao this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@AdrianoDee
Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild
Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49419/46877

@AdrianoDee
Copy link
Copy Markdown
Contributor Author

please abort

@cmsbuild
Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49419/46905

@cmsbuild
Copy link
Copy Markdown
Contributor

@AdrianoDee
Copy link
Copy Markdown
Contributor Author

please test
(with sentiment)


from Configuration.Eras.Era_Run3_2025_FastSim_cff import Run3_2025_FastSim

Run3_2026_FastSim = cms.ModifierChain(Run3_2025_FastSim)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to point out that all other Run*_FastSim eras are defined starting from Run*. Is the deviation from that pattern now intentional? (e.g. betting that Run3_2026 will always be the same as Run3_2025)

Copy link
Copy Markdown
Contributor Author

@AdrianoDee AdrianoDee Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, this is just a placeholder here. I would ask @cms-sw/simulation-l2 for suggestions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine for now. I actually hope we will be able to eliminate the Run3_202X_FastSim Eras entirely in the next couple of months, if the GEM developments converge. We will propagate the appropriate updates at that time.

@perrotta
Copy link
Copy Markdown
Contributor

  • auto:phase1_2026_* symbolic GTs are copies of 2025 ones (@cms-sw/alca-l2 let me know what you think about this, I'll updated them following your suggestion);

Just to let you know that, as for now, those are the only GTs available that can be used for this purpose. They can be replaced by the dedicated GTs with 2026 conditions as soon as they will become available

@AdrianoDee
Copy link
Copy Markdown
Contributor Author

I suspect the AMD_w7900 tests are stuck. For this specific PR, maybe we can skip them.
(@smuzaffar, can you give the bot a push on your side eventually? Thanks.)

@smuzaffar
Copy link
Copy Markdown
Contributor

I have restarted the comparison job. results should be available in few minutes

@cmsbuild
Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 100KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd870d/49598/summary.html
COMMIT: 1f2a5a4
CMSSW: CMSSW_16_0_X_2025-11-21-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 4 differences found in the comparisons
  • Reco comparison had 2 failed jobs
  • DQMHistoTests: Total files compared: 51
  • DQMHistoTests: Total histograms compared: 3905633
  • DQMHistoTests: Total failures: 3
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3905610
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
  • Checked 218 log files, 188 edm output root files, 51 DQM output files
  • TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

  • You potentially removed 5 lines from the logs
  • Reco comparison results: 238 differences found in the comparisons
  • Reco comparison had 6 failed jobs
  • DQMHistoTests: Total files compared: 11
  • DQMHistoTests: Total histograms compared: 148187
  • DQMHistoTests: Total failures: 27705
  • DQMHistoTests: Total nulls: 8
  • DQMHistoTests: Total successes: 120474
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
  • Checked 42 log files, 45 edm output root files, 11 DQM output files
  • TriggerResults: found differences in 1 / 10 workflows

AMD_W7900 Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • Reco comparison results: 245 differences found in the comparisons
  • Reco comparison had 6 failed jobs
  • DQMHistoTests: Total files compared: 11
  • DQMHistoTests: Total histograms compared: 148187
  • DQMHistoTests: Total failures: 28929
  • DQMHistoTests: Total nulls: 8
  • DQMHistoTests: Total successes: 119250
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
  • Checked 42 log files, 45 edm output root files, 11 DQM output files
  • TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

  • You potentially added 5 lines to the logs
  • Reco comparison results: 240 differences found in the comparisons
  • Reco comparison had 6 failed jobs
  • DQMHistoTests: Total files compared: 11
  • DQMHistoTests: Total histograms compared: 148187
  • DQMHistoTests: Total failures: 25798
  • DQMHistoTests: Total nulls: 7
  • DQMHistoTests: Total successes: 122382
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
  • Checked 42 log files, 45 edm output root files, 11 DQM output files
  • TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 211 differences found in the comparisons
  • Reco comparison had 6 failed jobs
  • DQMHistoTests: Total files compared: 11
  • DQMHistoTests: Total histograms compared: 148187
  • DQMHistoTests: Total failures: 33434
  • DQMHistoTests: Total nulls: 4
  • DQMHistoTests: Total successes: 114749
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
  • Checked 42 log files, 45 edm output root files, 11 DQM output files
  • TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

  • You potentially added 9 lines to the logs
  • Reco comparison results: 219 differences found in the comparisons
  • Reco comparison had 6 failed jobs
  • DQMHistoTests: Total files compared: 11
  • DQMHistoTests: Total histograms compared: 148187
  • DQMHistoTests: Total failures: 33731
  • DQMHistoTests: Total nulls: 7
  • DQMHistoTests: Total successes: 114449
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
  • Checked 42 log files, 45 edm output root files, 11 DQM output files
  • TriggerResults: found differences in 1 / 10 workflows

@AdrianoDee
Copy link
Copy Markdown
Contributor Author

+pdmv

@perrotta
Copy link
Copy Markdown
Contributor

+alca

@@ -14,6 +14,7 @@
'relval2023' : 'Fake2',
'relval2024' : 'Fake2',
'relval2025' : '2025v13',
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eventually, now that we have coverage I guess we can demote this to fake (in another PR).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'd wait to do that after pre3 is out just to use 2025 for one last round of validations.

@mmusich
Copy link
Copy Markdown
Contributor

mmusich commented Nov 24, 2025

+hlt

@cmsbuild
Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

@mmusich
Copy link
Copy Markdown
Contributor

mmusich commented Nov 24, 2025

Out of curiosity @cms-sw/dqm-l2 is it expected that in this PR the DQM bin-by-bin comparison is empty: https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqm-histo-comparison-summary.html ?

@gabrielmscampos
Copy link
Copy Markdown
Member

gabrielmscampos commented Nov 24, 2025

Out of curiosity @cms-sw/dqm-l2 is it expected that in this PR the DQM bin-by-bin comparison is empty: cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqm-histo-comparison-summary.html ?

I'd say that is unusual. I was taking a look at signed and merged PRs from last week that target 16_0_X and the cmsdt pages are still available and most of them also are displaying empty DQM bin-by-bin comparison.

On this specific PR, I cannot find the /RelVal_wf25202_0_pr/CMSSW_16_0_X-PRcmssw_49419-XXXXXX/DQMIO dataset in the Development GUI. Maybe the ROOT file was not uploaded at the end of the tests? Can we check Jenkins's logs to check this?

@mmusich
Copy link
Copy Markdown
Contributor

mmusich commented Nov 24, 2025

shall we try to test again?

@smuzaffar
Copy link
Copy Markdown
Contributor

smuzaffar commented Nov 24, 2025

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqmBinByBinLog.log shows some root/python errors [a] . Could this be the reason the dqm bin-by-bin comparison were not properly done ? Could it be due to ROOT 6.36 update?

[a]

PR file successfully opened
Traceback (most recent call last):
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 345, in <module>
    create_dif(args.base_file, args.pr_file, args.pr_number, args.test_number, cmssw_version, args.num_processes, args.output_dir)
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 80, in create_dif
    compare(shared_paths, pr_flat_dict, base_flat_dict, paths_to_save_in_pr, paths_to_save_in_base)
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 182, in compare
    pr_array = np.array(pr_item)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 367, in _getitem
    uhi_index = _compute_common_index(self, index)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 223, in _compute_common_index
    return [_compute_uhi_index(self, idx, axis, include_flow_bins) for axis, idx in enumerate(index)]
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 223, in <listcomp>
    return [_compute_uhi_index(self, idx, axis, include_flow_bins) for axis, idx in enumerate(index)]
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 192, in _compute_uhi_index
    return _process_index_for_axis(self, index, axis)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 180, in _process_index_for_axis
    raise IndexError(f"Histogram index {index - 1} out of range for axis {axis}. Valid range: (0,{nbins})")
IndexError: Histogram index 30 out of range for axis 0. Valid range: (0,30)

@gabrielmscampos
Copy link
Copy Markdown
Member

cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqmBinByBinLog.log shows some root/python errors [a] . Could this be the reason the dqm bin-by-bin comparison were not properly done ? Could it be due to ROOT 6.36 update?

That is right, at the end of the log we can see that there were no files to be uploaded as well.

shall we try to test again?

I don't think testing again will solve the issue.

@mmusich
Copy link
Copy Markdown
Contributor

mmusich commented Nov 24, 2025

Could this be the reason the dqm bin-by-bin comparison were not properly done ? Could it be due to ROOT 6.36 update?

so just to be clear, the issue is not coming specifically from this PR, but it's also present in all other recent PRs?

@gabrielmscampos
Copy link
Copy Markdown
Member

@smuzaffar
Copy link
Copy Markdown
Contributor

this might be related to ROOT 6.36 change root-project/root@ff65e3b which is in 16.0.X IBs .

@smuzaffar
Copy link
Copy Markdown
Contributor

@smuzaffar
Copy link
Copy Markdown
Contributor

I am opening a new issue to track dqm bin-by-bin failures

@smuzaffar
Copy link
Copy Markdown
Contributor

#49452

@mmusich
Copy link
Copy Markdown
Contributor

mmusich commented Nov 24, 2025

I am opening a new issue to track dqm bin-by-bin failures

thanks!

@mandrenguyen
Copy link
Copy Markdown
Contributor

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants