Fixed Number of Events Data Wfs for Testing by AdrianoDee · Pull Request #48490 · cms-sw/cmssw

AdrianoDee · 2025-07-07T10:26:03Z

PR description:

This PR proposes to move back the Run3 data wfs used for testing to be the the YYYY.* ones (with YYYY being the data taking year). These run the fixed number of events selection machinery. To do this the das-up-to-nevents.py script has been updated in order to run only two very small dasgoclient queries [1], instead of few big ones or many small (with -pc), if it's run within the PR or IBs tests (checking if the environmental variable JENKINS_PREFIX is set). I took also the chance to:

clean up a bit the machinery to avoid code duplications;
move all 2024 data wfs to use non-prompt conditions for offline;
add 2025.* wfs to the limited matrix;
include the update to the hlt key in Moving 2024 Data Wfs HLT Key to hltKey2024 #48479 (so moving back 2024 wfs to run HLT:@relval2024);
add HFlavor DQM for ParkingDoubleMuonLowMass.

PR validation:

runTheMatrix -l run3

[1]

Running DAS command: dasgoclient --query='run dataset=/Muon0/Run2025B-v1/RAW  '
['391548', '391549', '391550', '391551', '391560', '391565', '391572', '391580', '391600', '391601', '391602', '391604', '391658', '391668', '391670', '391671', '391673', '391674', '391675', '391676', '391677', '391678', '391679', '391682', '391683', '391684', '391685', '391686', '391687', '391688', '391698', '391699', '391700', '391787', '391825', '391826', '391869', '391870', '391871', '391873', '391874', '391883', '391884', '391906', '391908', '391909', '391910', '391912', '391949', '391950', '391951', '391952', '391953', '392030', '392046', '392052', '392067', '392071', '392077', '392086', '392088', '392089', '392090', '392091', '392092', '392094', '392109', '392112', '']
Running DAS command: dasgoclient --query='file,lumi,run dataset=/Muon0/Run2025B-v1/RAW run=391668'
['/store/data/Run2025B/Muon0/RAW/v1/000/391/668/00000/311b85c3-fad4-488b-8209-2c7b17d0c1ec.root 391668 [30,36,37,49,53,65,66,83,85,91,94,108,117,124,127,151,155,156,158,168,172,192,193,205,208,225,227,240,246,252,254,255,257,263,266,14,16,19,32,41,43,45,58,63,74,77,78,82,105,123,134,149,152,183,197,204,207,226,250,12,15,18,20,25,42,47,54,67,69,70,80,88,98,111,116,120,125,126,141,144,147,148,179,181,194,198,202,213,217,230,231,232,234,261,264,8,10,17,27,29,40,50,52,96,103,104,112,130,131,132,133,157,161,163,166,173,175,182,184,188,189,200,235,247,248,253,258,260,262,11,21,23,26,33,38,39,51,56,57,60,64,68,84,86,99,101,106,110,113,119,137,138,139,142,143,146,154,159,164,167,195,199,218,237,238,239,241,243,244,267,13,24,28,35,59,62,71,73,118,128,162,170,171,178,190,191,196,203,206,211,212,216,219,222,229,242,256,259,265,7,9,34,46,48,55,61,79,87,92,93,95,114,115,129,136,145,150,160,165,174,176,186,187,201,209,210,221,224,228,249,22,31,44,72,75,76,81,89,90,97,100,102,107,109,121,122,135,140,153,169,177,180,185,214,215,220,223,233,236,245,251]', '']
/store/data/Run2025B/Muon0/RAW/v1/000/391/668/00000/311b85c3-fad4-488b-8209-2c7b17d0c1ec.root

cmsbuild · 2025-07-07T10:26:23Z

cms-bot internal usage

cmsbuild · 2025-07-07T10:27:36Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48490/45422

cmsbuild · 2025-07-07T10:28:01Z

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

Configuration/PyReleaseValidation (upgrade, pdmv)

@AdrianoDee, @Moanwar, @cmsbuild, @DickyChant, @miquork, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @slomeo this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2025-07-07T10:30:43Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48490/45423

cmsbuild · 2025-07-07T10:31:05Z

Pull request #48490 was updated. @AdrianoDee, @Moanwar, @cmsbuild, @DickyChant, @miquork, @srimanob, @subirsarkar can you please check and sign again.

AdrianoDee · 2025-07-07T10:31:44Z

please test

cmsbuild · 2025-07-07T12:09:54Z

-1

Failed Tests: RelVals RelVals-INPUT
Size: This PR adds an extra 100KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7a4189/47108/summary.html
COMMIT: 64cacda
CMSSW: CMSSW_15_1_X_2025-07-07-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48490/47108/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals

2023.0020001

----- Begin Fatal Exception 07-Jul-2025 12:50:16 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2023D/JetMET0/RAW/v1/000/369/927/00000/fd260ebe-58fa-4e21-958d-7e6916b63bda.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2023D/JetMET0/RAW/v1/000/369/927/00000/fd260ebe-58fa-4e21-958d-7e6916b63bda.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2023D/JetMET0/RAW/v1/000/369/927/00000/fd260ebe-58fa-4e21-958d-7e6916b63bda.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2023D/JetMET0/RAW/v1/000/369/927/00000/fd260ebe-58fa-4e21-958d-7e6916b63bda.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2023D/JetMET0/RAW/v1/000/369/927/00000/fd260ebe-58fa-4e21-958d-7e6916b63bda.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2023D/JetMET0/RAW/v1/000/369/927/00000/fd260ebe-58fa-4e21-958d-7e6916b63bda.root?tried=+1213xrootd-redic.pi.infn.it&xrdcl.requuid=1026a38e-a52e-41c5-9bd6-558786c3a835
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

2024.0030001DAS Error
2024.0010001DAS Error

Expand to see more relval errors ...

2024.0070001
2024.0050001
2024.0020001
2024.0000001
2024.0060001
2024.0040001

RelVals-INPUT

2025.0060001DAS Error
2023.00200012023.0020001_RunJetMET02023D_10k/step2_RunJetMET02023D_10k.log
2025.0020001DAS Error

Expand to see more relval errors ...

2024.0030001
2024.0010001
2024.0070001
2025.0040001
2024.0050001
2025.0070001
2025.0050001
2025.0030001
2022.0000001
2022.0020001
2024.0020001
2024.0000001
2024.0060001
2024.0040001

AdrianoDee · 2025-09-16T12:36:58Z

please test
(the file should be finally on disk)

cmsbuild · 2025-09-16T14:14:14Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7a4189/48158/summary.html
COMMIT: 1efc5f0
CMSSW: CMSSW_16_0_X_2025-09-15-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48490/48158/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 40
DQMHistoTests: Total histograms compared: 3274387
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3274364
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 39 files compared)
Checked 175 log files, 144 edm output root files, 40 DQM output files
TriggerResults: no differences found

AdrianoDee · 2025-09-16T14:16:14Z

+pdmv

AdrianoDee · 2025-09-19T09:55:03Z

@cms-sw/upgrade-l2 sorry to bother: could you have a look?

Moanwar · 2025-09-19T10:14:32Z

+Upgrade

cmsbuild · 2025-09-19T10:14:54Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2025-09-20T07:37:57Z

+1

mandrenguyen · 2025-09-29T08:09:49Z

(just cross posting)
There are frequent DAS failures over the past week that seem to point back to this PR, e.g., #49014 (comment)
Would @cms-sw/pdmv-l2 kindly has a look?

AdrianoDee · 2025-09-29T08:11:03Z

Would @cms-sw/pdmv-l2 kindly has a look?

Sure, I'm having a look.

AdrianoDee · 2025-09-29T10:44:03Z

Sure, I'm having a look.

#49020 will solve, as soon as the files are copied on disk.

cmsbuild added this to the CMSSW_15_1_X milestone Jul 7, 2025

cmsbuild added pending-signatures tests-pending orp-pending pdmv-pending upgrade-pending code-checks-pending labels Jul 7, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 7, 2025

AdrianoDee force-pushed the fixed_data_wfs_for_testing branch from c49c8f6 to 8b671f2 Compare July 7, 2025 10:28

cmsbuild added code-checks-pending and removed code-checks-approved labels Jul 7, 2025

AdrianoDee force-pushed the fixed_data_wfs_for_testing branch from 8d9895a to 64cacda Compare July 7, 2025 10:29

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 7, 2025

cmsbuild added tests-started and removed tests-pending labels Jul 7, 2025

AdrianoDee mentioned this pull request Jul 7, 2025

Moving 2024 Data Wfs HLT Key to hltKey2024 #48479

Closed

cmsbuild added tests-rejected and removed tests-started labels Jul 7, 2025

AdrianoDee force-pushed the fixed_data_wfs_for_testing branch from 64cacda to edb3e28 Compare July 7, 2025 13:31

cmsbuild removed tests-rejected code-checks-approved labels Jul 7, 2025

This was referenced Sep 12, 2025

[NGT] Extension of CA Pixel Tracking to Phase 2 Outer Tracker barrel #48921

Merged

remove wf 24834.911 #48925

Merged

cmsbuild added tests-started and removed tests-rejected labels Sep 16, 2025

cmsbuild added tests-approved and removed tests-started labels Sep 16, 2025

cmsbuild added pdmv-approved and removed pdmv-pending labels Sep 16, 2025

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels Sep 19, 2025

cmsbuild added orp-approved and removed orp-pending labels Sep 20, 2025

cmsbuild merged commit 73d3859 into cms-sw:master Sep 20, 2025
10 checks passed

dan131riley mentioned this pull request Sep 22, 2025

Relval 145.415 step 2: requested pattern "DST_PFScouting_ZeroBias*" does not match any HLT paths #48964

Closed

cmsbuild mentioned this pull request Sep 22, 2025

[ROOT636] Root update v6 36 00 patches 20250922 cms-sw/cmsdist#10087

Closed

mmusich mentioned this pull request Sep 29, 2025

PrimaryVertexResolution: add protection against missing track references in input vertex #49014

Merged

AdrianoDee mentioned this pull request Sep 29, 2025

Simplify Data Workflows in Tests (No JSON Check) #49020

Closed

AdrianoDee mentioned this pull request Oct 2, 2025

Reading Certification JSON from cvmfs in das-up-to-nevents.py #49051

Merged

Conversation

AdrianoDee commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Uh oh!

cmsbuild commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

AdrianoDee commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

RelVals

RelVals-INPUT

Uh oh!

AdrianoDee commented Sep 16, 2025

Uh oh!

cmsbuild commented Sep 16, 2025

Comparison Summary

Uh oh!

AdrianoDee commented Sep 16, 2025

Uh oh!

AdrianoDee commented Sep 19, 2025

Uh oh!

Moanwar commented Sep 19, 2025

Uh oh!

cmsbuild commented Sep 19, 2025

Uh oh!

mandrenguyen commented Sep 20, 2025

Uh oh!

Uh oh!

mandrenguyen commented Sep 29, 2025

Uh oh!

AdrianoDee commented Sep 29, 2025

Uh oh!

AdrianoDee commented Sep 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AdrianoDee commented Jul 7, 2025 •

edited

Loading

cmsbuild commented Jul 7, 2025 •

edited

Loading