2025 Wfs Skeleton and Updates to Data Wfs (2024,2023,2022) by AdrianoDee · Pull Request #46445 · cms-sw/cmssw

AdrianoDee · 2024-10-18T14:24:37Z

PR description:

This PR proposes the addition of 2025 wfs to the matrix. For the moment they are a copy of what we run currently for 2024 with the idea that once 2025 parameters (conditions, HLT) come we will simply need to fill the dummy ones (keeping them in sync one with the other). Here I add also:

@relval2025 pointing to GRun;
auto:phase1_2025_* symbolic GTs;

The era is Era_2025 that for now is simply a copy of Era_2024. A TTbar 2025 noPU wf is added to the limited matrix to be tested with PRs.

The rationale is that I would like to avoid (with some anticipation) that we fall again in a situation such as #44028 or #41410.

Additional changes:

2023 data wfs GT moved to the offline one (auto:run3_data_relval);
2024 data wfs added to the PR tests reducing the number of 2022,2023 wfs;
small fixes for ZeroBias 2024 wfs to have the RECO datatier produced for them (as it was in 2023 and 2022) and to run the proper DQM sequence.
2024 HLT moved to relval2025 to keep GRun tested;
relval_2024 moved to Fake2;
2023 data wfs HLT moved to relval_2023 (Fake2);

PR validation:

New wfs run.

cmsbuild · 2024-10-18T14:24:52Z

@AdrianoDee, CMSSW_14_2_X branch is closed for direct updates. cms-bot is going to move this PR to master branch.
In future, please use cmssw master branch to submit your changes.

cmsbuild · 2024-10-18T14:25:10Z

cms-bot internal usage

AdrianoDee · 2024-10-18T14:25:16Z

please test

cmsbuild · 2024-10-18T14:27:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46445/42304

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): New Inputs for D110 And Enabling HIN Phase2 Wf #46355
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, CMSSW Integration of LST #45117, Introduce support for cluster splitting and JetCore iteration in Phase-2 track reconstruction #46001

cmsbuild · 2024-10-18T14:27:55Z

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

Configuration/AlCa (alca)
Configuration/HLT (hlt)
Configuration/PyReleaseValidation (upgrade, pdmv)

@AdrianoDee, @Martin-Grunewald, @Moanwar, @atpathak, @consuegs, @kskovpen, @miquork, @mmusich, @perrotta, @srimanob, @subirsarkar, @sunilUIET can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @mmusich, @rsreds, @silviodonato, @slomeo, @tocheng, @yuanchao this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2024-10-18T14:30:47Z

@@ -13,6 +13,7 @@
  'relval2022' : 'Fake2',
  'relval2023' : 'Fake2',
  'relval2024' : 'GRun',


are we going to get 2025 relvals to run in IBs and PRs?

IBs no, PRs I've added a single ttbar noPu.

(explaining myself, if we overhaul completely 2024 in favor of 2025, we can demote relval2024 to Fake2 in this cycle as the purpose of having GRun here is to keep having workflows running it somewhere)

Meaning: I didn't add the in this PR but it would be doable.

(explaining myself, if we overhaul completely 2024 in favor of 2025, we can demote relval2024 to Fake2 in this cycle as the purpose of having GRun here is to keep having workflows running it somewhere)

Then yes we could do that.

we're not going to re-run the 2024 HLT in any other release than 14.0.X in production. I don't see the reason to validate the 2024 HLT in this cycle. Of course "offline" validators can continue to validate offline reconstruction.

This is clear, I wanted to understand if it's anyway useful to keep an eye on the 2024 GT (for the "offline" validations).

If it's not, I'll move everything to 2025 and that's it.

I've moved @relval2024 to Fake and added a couple of 2025 wfs to IB tests.

This is clear, I wanted to understand if it's anyway useful to keep an eye on the 2024 GT (for the "offline" validations).

2024 pp GTs are supposed to be "completed" by now.
There is still some work in progress for the 2024 HI GTs, but they are validated via different workflows than the ones addressed here.

cmsbuild · 2024-10-18T15:24:22Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 92KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-758440/42298/summary.html
COMMIT: e6ad2e9
CMSSW: CMSSW_14_2_X_2024-10-17-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46445/42298/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

ValueError: Undefined workflows: 16834.0

AdrianoDee · 2024-10-19T17:35:09Z

please test

cmsbuild · 2024-10-28T09:28:03Z

Pull request #46445 was updated. @AdrianoDee, @Martin-Grunewald, @Moanwar, @atpathak, @cmsbuild, @consuegs, @kskovpen, @miquork, @mmusich, @perrotta, @srimanob, @subirsarkar, @sunilUIET can you please check and sign again.

AdrianoDee · 2024-10-28T09:28:47Z

please test

cmsbuild · 2024-10-28T11:35:20Z

-1

Failed Tests: RelVals RelVals-INPUT
Size: This PR adds an extra 104KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-758440/42413/summary.html
COMMIT: c9e7be9
CMSSW: CMSSW_14_2_X_2024-10-27-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46445/42413/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

2023.002001

----- Begin Fatal Exception 28-Oct-2024 11:07:43 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2023D/ZeroBias/RAW/v1/000/369/927/00000/2c0c87bd-ba37-4e2c-9211-098c2efcdf2b.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2023D/ZeroBias/RAW/v1/000/369/927/00000/2c0c87bd-ba37-4e2c-9211-098c2efcdf2b.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2023D/ZeroBias/RAW/v1/000/369/927/00000/2c0c87bd-ba37-4e2c-9211-098c2efcdf2b.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2023D/ZeroBias/RAW/v1/000/369/927/00000/2c0c87bd-ba37-4e2c-9211-098c2efcdf2b.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2023D/ZeroBias/RAW/v1/000/369/927/00000/2c0c87bd-ba37-4e2c-9211-098c2efcdf2b.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2023D/ZeroBias/RAW/v1/000/369/927/00000/2c0c87bd-ba37-4e2c-9211-098c2efcdf2b.root?tried=+1213xrootd-redic.pi.infn.it,&xrdcl.requuid=7c4d6db8-4559-4619-ad02-d9e203f20f45
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

2024.000001

----- Begin Fatal Exception 28-Oct-2024 11:08:17 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2024B/ZeroBias/RAW/v1/000/378/985/00000/11681899-5f48-4e26-949d-30e452ae1ba6.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2024B/ZeroBias/RAW/v1/000/378/985/00000/11681899-5f48-4e26-949d-30e452ae1ba6.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2024B/ZeroBias/RAW/v1/000/378/985/00000/11681899-5f48-4e26-949d-30e452ae1ba6.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2024B/ZeroBias/RAW/v1/000/378/985/00000/11681899-5f48-4e26-949d-30e452ae1ba6.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2024B/ZeroBias/RAW/v1/000/378/985/00000/11681899-5f48-4e26-949d-30e452ae1ba6.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2024B/ZeroBias/RAW/v1/000/378/985/00000/11681899-5f48-4e26-949d-30e452ae1ba6.root?tried=+1213xrootd-cms-redir-int.cr.cnaf.infn.it,&xrdcl.requuid=a5303e39-c857-489d-abd5-eeb00713aab7
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

2023.0020012023.002001_RunZeroBias2023D_10k/step2_RunZeroBias2023D_10k.log
2024.0000012024.000001_RunZeroBias2024B_10k/step2_RunZeroBias2024B_10k.log

Martin-Grunewald · 2024-10-29T12:56:40Z

-  'relval2024' : 'GRun',
+  'relval2024' : 'Fake2',
+  'relval2025' : 'GRun',
  'relval2026' : '75e33',


Hmm, 2026 is rather confusing by now as it is by now still phase1, rather than phase2 which should now be 2030?
When is that update redefining 2026 foreseen?

Indeed, it is the next item on my todo list.

RelVals

2024.0000012024.000001_RunZeroBias2024B_10k/step1_dasquery.log
2024.1010012024.101001_RunBTagMu2024C_10k/step1_dasquery.log
2024.2020012024.202001_RunJetMET02024D_10k/step1_dasquery.log

Expand to see more relval errors ...

2024.303001
2024.000001
2024.101001
2024.202001
2024.303001

RelVals-INPUT

2024.0000012024.000001_RunZeroBias2024B_10k/step1_dasquery.log
2024.0010012024.001001_RunBTagMu2024B_10k/step1_dasquery.log
2024.0020012024.002001_RunJetMET02024B_10k/step1_dasquery.log

Expand to see more relval errors ...

2024.003001
2024.100001
2024.101001
2024.102001
2024.103001
2024.200001
2024.201001
2024.202001
2024.203001
2024.300001
2024.301001
2024.302001
2024.303001
2024.000001
2024.001001
2024.002001
2024.003001
2024.100001
2024.101001
2024.102001
2024.103001
2024.200001
2024.201001
2024.202001
2024.203001
2024.300001
2024.301001
2024.302001
2024.303001

AdrianoDee · 2024-10-30T20:47:10Z

please test with #46559

cmsbuild · 2024-10-31T02:34:16Z

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-758440/42487/summary.html
COMMIT: c9e7be9
CMSSW: CMSSW_14_2_X_2024-10-30-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46445/42487/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 157 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 3543 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2641417
DQMHistoTests: Total failures: 65297
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2576100
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -700067.382 KiB( 37 files compared)
DQMHistoSizes: changed ( 12834.0,... ): -43568.391 KiB HLT/JME
DQMHistoSizes: changed ( 12834.0,... ): -17546.239 KiB HLT/Tracking
DQMHistoSizes: changed ( 12834.0,... ): -16130.179 KiB HLT/BTV
DQMHistoSizes: changed ( 12834.0,... ): -14352.967 KiB HLT/EGM
DQMHistoSizes: changed ( 12834.0,... ): -10467.421 KiB HLT/EXO
DQMHistoSizes: changed ( 12834.0,... ): -10390.469 KiB HLT/SiStrip
DQMHistoSizes: changed ( 12834.0,... ): -10237.354 KiB HLT/HCAL
DQMHistoSizes: changed ( 12834.0,... ): -9248.691 KiB HLT/Vertexing
DQMHistoSizes: changed ( 12834.0,... ): -9093.510 KiB HLT/Muon
DQMHistoSizes: changed ( 12834.0,... ): -9066.871 KiB HLT/Pixel
DQMHistoSizes: changed ( 12834.0 ): ...
Checked 169 log files, 139 edm output root files, 38 DQM output files
TriggerResults: found differences in 4 / 36 workflows

AdrianoDee · 2024-10-31T05:20:42Z

+pdmv

mmusich · 2024-10-31T07:53:04Z

+hlt

for AdrianoDee@c9e7be9
PR according to description and follow-up review

perrotta · 2024-10-31T09:56:34Z

            wf_number = wf_number + offset_era * e_n
            wf_number = wf_number + offset_pd * p_n
-            wf_number = wf_number + offset_events * evs 
+            wf_number = wf_number + 0.0001 * 0.01 


Just curious: why did you move to the hardcoded numbers rather than sticking on the parameterized ones? Why do you do so for all eras but the eras_2022_2 one?

It was a stupid copy paste. Thanks for spotting it. Just reverted.

AdrianoDee · 2024-10-31T10:27:29Z

please test with #46559

cmsbuild · 2024-10-31T10:29:10Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46445/42447

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, CMSSW Integration of LST #45117, ECAL RecHit producer Alpaka migration #46453, Remove PU from ProdLike Wfs #46521

cmsbuild · 2024-10-31T10:29:38Z

Pull request #46445 was updated. @AdrianoDee, @Moanwar, @atpathak, @consuegs, @DickyChant, @miquork, @perrotta, @srimanob, @subirsarkar can you please check and sign again.

AdrianoDee · 2024-10-31T10:30:04Z

please abort

AdrianoDee · 2024-10-31T10:44:24Z

please test with #46559

cmsbuild · 2024-10-31T19:02:46Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-758440/42503/summary.html
COMMIT: 7f14a11
CMSSW: CMSSW_14_2_X_2024-10-30-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46445/42503/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 162 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 3539 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2641417
DQMHistoTests: Total failures: 65273
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2576123
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -700067.382 KiB( 37 files compared)
DQMHistoSizes: changed ( 12834.0,... ): -43568.391 KiB HLT/JME
DQMHistoSizes: changed ( 12834.0,... ): -17546.239 KiB HLT/Tracking
DQMHistoSizes: changed ( 12834.0,... ): -16130.179 KiB HLT/BTV
DQMHistoSizes: changed ( 12834.0,... ): -14352.967 KiB HLT/EGM
DQMHistoSizes: changed ( 12834.0,... ): -10467.421 KiB HLT/EXO
DQMHistoSizes: changed ( 12834.0,... ): -10390.469 KiB HLT/SiStrip
DQMHistoSizes: changed ( 12834.0,... ): -10237.354 KiB HLT/HCAL
DQMHistoSizes: changed ( 12834.0,... ): -9248.691 KiB HLT/Vertexing
DQMHistoSizes: changed ( 12834.0,... ): -9093.510 KiB HLT/Muon
DQMHistoSizes: changed ( 12834.0,... ): -9066.871 KiB HLT/Pixel
DQMHistoSizes: changed ( 12834.0 ): ...
Checked 169 log files, 139 edm output root files, 38 DQM output files
TriggerResults: found differences in 4 / 36 workflows

AdrianoDee · 2024-10-31T20:24:35Z

+pdmv

perrotta · 2024-10-31T21:21:17Z

+alca

Moanwar · 2024-11-03T15:14:27Z

+Upgrade

cmsbuild · 2024-11-03T15:14:55Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2024-11-04T10:15:03Z

+1

AdrianoDee marked this pull request as ready for review October 18, 2024 14:24

cmsbuild changed the base branch from CMSSW_14_2_X to master October 18, 2024 14:24

cmsbuild added this to the CMSSW_14_2_X milestone Oct 18, 2024

cmsbuild added hlt-pending alca-pending pending-signatures tests-pending orp-pending pdmv-pending upgrade-pending code-checks-pending labels Oct 18, 2024

cmsbuild added tests-started and removed tests-pending labels Oct 18, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 18, 2024

mmusich reviewed Oct 18, 2024

View reviewed changes

cmsbuild added tests-rejected tests-pending code-checks-pending and removed tests-started tests-rejected code-checks-approved labels Oct 18, 2024

AdrianoDee force-pushed the 2025_wfs branch from e4df2ee to 49a82d8 Compare October 19, 2024 17:31

Martin-Grunewald reviewed Oct 29, 2024

View reviewed changes

cmsbuild mentioned this pull request Oct 29, 2024

CMSSW Integration of LST #45117

Merged

3 tasks

perrotta reviewed Oct 31, 2024

View reviewed changes

Fixing data wfs, updating matrix

7f14a11

This was referenced Nov 1, 2024

Rename Phase-2 "2026" geometries and workflows to "Run4" #46582

Merged

Add WF for 2024 PbPb skim #46589

Merged

cmsbuild mentioned this pull request Nov 4, 2024

Fixing Wf Names for ParkingDouble* #46601

Closed

AdrianoDee mentioned this pull request Nov 19, 2025

2026 RelVal Workflows #49419

Merged

+              steps['HLTDR3_2023B']=merge( [ {'-s':'L1REPACK:Full,HLT:@%s'%hltKey2023,},{'--conditions':'auto:run3_hlt_relval'},{'--era' : 'Run3'},steps['HLTD'] ] )
-              steps['HLTDR3_2024']=merge( [ {'-s':'L1REPACK:Full,HLT:@%s'%hltKey2024,},{'--conditions':'auto:run3_hlt_relval'},{'--era' : 'Run3_2024'},steps['HLTD'] ] )
+              steps['HLTDR3_2024']=merge( [ {'-s':'L1REPACK:Full,HLT:@%s'%hltKey2025,},{'--conditions':'auto:run3_hlt_relval'},{'--era' : 'Run3_2024'},steps['HLTD'] ] )

Conversation

AdrianoDee commented Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Uh oh!

cmsbuild commented Oct 18, 2024

Uh oh!

cmsbuild commented Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdrianoDee commented Oct 18, 2024

Uh oh!

cmsbuild commented Oct 18, 2024

Uh oh!

cmsbuild commented Oct 18, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmusich Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented Oct 18, 2024

RelVals

Uh oh!

AdrianoDee commented Oct 19, 2024

Uh oh!

cmsbuild commented Oct 28, 2024

Uh oh!

AdrianoDee commented Oct 28, 2024

Uh oh!

cmsbuild commented Oct 28, 2024

RelVals

RelVals-INPUT

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AdrianoDee commented Oct 29, 2024

Uh oh!

AdrianoDee commented Oct 30, 2024

Uh oh!

cmsbuild commented Oct 30, 2024

RelVals

RelVals-INPUT

Uh oh!

AdrianoDee commented Oct 30, 2024

Uh oh!

cmsbuild commented Oct 31, 2024

Comparison Summary

Uh oh!

AdrianoDee commented Oct 31, 2024

Uh oh!

mmusich commented Oct 31, 2024

Uh oh!

Choose a reason for hiding this comment

AdrianoDee commented Oct 18, 2024 •

edited

Loading

cmsbuild commented Oct 18, 2024 •

edited

Loading

mmusich Oct 18, 2024 •

edited

Loading