Simplify Data Workflows in Tests (No JSON Check) by AdrianoDee · Pull Request #49020 · cms-sw/cmssw

AdrianoDee · 2025-09-29T10:42:54Z

PR description:

Lately (with #48490), spurious DAS errors appear frequently in the tests (see e.g. #49014 (comment)). This is due to random glitches in accessing https://cms-service-dqmdc.web.cern.ch/CAF/certification/ when running the tests (no access to EOS is available). This results in not finding any certification JSON files to use for skimming the datasets. One possibility could be to allow for a few retries in case of failures, but I find it anyway risky.

So this PR proposes simply to skip the golden JSON check when running the tests (so when JENKINS_PREFIX is defined).

I'm opening this as a draft just until the files are copied to CERN.

e286032666c249b4a4e76dadba0a3f68  pdmv_group  cms:/store/data/Run2022E/BTagMu/RAW/v1/000/359/037/00000/b101e8c5-4d7f-4853-896d-77967dff5ce6.root                                                                                      REPLICATING[0/1/0]      T2_CH_CERN        1         1.319 MB                     2025-09-29 09:19:12
d10878e0f790434c8ca656337a888d5e  pdmv_group  cms:/store/data/Run2023D/JetMET0/RAW/v1/000/369/869/00000/cf59ba02-cc61-4768-885f-b6b45baccfe2.root                                                                                     REPLICATING[0/1/0]      T2_CH_CERN        1         6.474 GB                     2025-09-29 10:06:22
09d38828b9b9419497e769ae5f9695b3  pdmv_group  cms:/store/data/Run2025B/ZeroBias/RAW/v1/000/391/531/00000/31f4860a-232d-43c0-aa51-e805866b0677.root                                                                                    REPLICATING[0/1/0]      T2_CH_CERN        1         70.210 MB                    2025-09-29 09:19:30
a289c7e9ddb247788c555d2c3222e47b  pdmv_group  cms:/store/data/Run2024C/JetMET0/RAW/v1/000/379/415/00000/41e6df86-f198-4b44-886e-5d68a8e8df1b.root                                                                                     REPLICATING[0/1/0]      T2_CH_CERN        1         6.010 GB                     2025-09-29 10:06:27
c62a2dc81ce1469ea7dda7c701c1a2b3  pdmv_group  cms:/store/data/Run2025C/JetMET0/RAW/v1/000/392/174/00000/f9fa84d2-905c-4ed2-8821-18a89675e4eb.root                                                                                     OK[1/0/0]               T2_CH_CERN        1         12.061 MB                    2025-09-29 09:19:32
699477364c664fcfa7a1389dc848024d  pdmv_group  cms:/store/data/Run2024E/DisplacedJet/RAW/v1/000/380/963/00000/7690f63b-c907-45b3-9ec9-fc5c9ad7c1ac.root                                                                                REPLICATING[0/1/0]      T2_CH_CERN        1         3.836 GB                     2025-09-29 10:06:36
bdb988b8473d4f19af41b44194b4fbb6  pdmv_group  cms:/store/data/Run2023D/JetMET0/RAW/v1/000/369/844/00000/333b74fc-a093-47dd-9d6c-a2373c690134.root                                                                                     REPLICATING[0/1/0]      T2_CH_CERN        1         1.544 MB                     2025-09-29 09:19:14
6db1bc89296a4ea4a9e4c587d7ec9c79  pdmv_group  cms:/store/data/Run2024E/DisplacedJet/RAW/v1/000/380/956/00000/140c01c0-1725-476e-adaa-37bbbde6a09d.root                                                                                REPLICATING[0/1/0]      T2_CH_CERN        1         1.343 MB                     2025-09-29 09:19:22
13a8ad84efec452495b7a9a6415a8d29  pdmv_group  cms:/store/data/Run2024I/Tau/RAW/v1/000/386/446/00000/b6577323-4ae5-45e7-b9d8-7f97fbc0cc15.root                                                                                         OK[1/0/0]               T2_CH_CERN        1         1.377 MB                     2025-09-29 09:19:28
6e3347c899894f44a53c5bb75244325e  pdmv_group  cms:/store/data/Run2024F/ParkingDoubleMuonLowMass0/RAW/v1/000/382/213/00000/13227a75-4eca-4f1a-8734-5c1e6bee9a82.root                                                                   OK[1/0/0]               T2_CH_CERN        1         3.201 GB                     2025-09-29 10:06:39
c2a08fcbcdff42c2b45df6c35c9314ea  pdmv_group  cms:/store/data/Run2024C/JetMET0/RAW/v1/000/379/413/00000/2c0fee08-8977-4e2a-b369-bfa7101aa2c8.root                                                                                     REPLICATING[0/1/0]      T2_CH_CERN        1         1.317 MB                     2025-09-29 09:19:19
779124754d9b42c2925605a9a77ffa1a  pdmv_group  cms:/store/data/Run2024B/ZeroBias/RAW/v1/000/378/981/00000/a27bb954-1259-470c-8874-4d94ff5df34a.root                                                                                    REPLICATING[0/1/0]      T2_CH_CERN        1         3.325 GB                     2025-09-29 09:19:17
520cf8dd550e4f5689113e5a1f6c485b  pdmv_group  cms:/store/data/Run2024F/ParkingDoubleMuonLowMass0/RAW/v1/000/382/209/00000/fa5a6a3c-6db8-42f7-baab-df1965111988.root                                                                   OK[1/0/0]               T2_CH_CERN        1         6.443 MB                     2025-09-29 09:19:24

PR validation:

Running the standard tests.

cmsbuild · 2025-09-29T10:43:23Z

cms-bot internal usage

cmsbuild · 2025-09-29T10:44:32Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49020/46220

AdrianoDee · 2025-09-30T09:38:45Z

please test

cmsbuild · 2025-09-30T09:39:06Z

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

Configuration/PyReleaseValidation (upgrade, pdmv)

@AdrianoDee, @DickyChant, @Moanwar, @antoniovagnerini, @miquork, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2025-09-30T12:10:33Z

-1

Failed Tests: RelVals-INPUT
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35e700/48372/summary.html
COMMIT: dda7573
CMSSW: CMSSW_16_0_X_2025-09-29-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49020/48372/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

2023.00000012023.0000001_RunZeroBias2023B_10k/step2_RunZeroBias2023B_10k.log
2022.00100012022.0010001_RunJetHT2022C_10k/step2_RunJetHT2022C_10k.log
2022.00000012022.0000001_RunZeroBias2022B_10k/step2_RunZeroBias2022B_10k.log

Comparison Summary

The workflows 2025.0010001, 2025.0000001, 2024.0040001, 2024.0030001, 2024.0020001, 2024.0010001, 2024.0000001, 2023.0020001 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

You potentially added 1691 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 18726 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3503512
DQMHistoTests: Total failures: 8536
DQMHistoTests: Total nulls: 46
DQMHistoTests: Total successes: 3494910
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 40.6 KiB( 44 files compared)
DQMHistoSizes: changed ( 2024.0020001 ): 2.363 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0020001 ): 2.320 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 2024.0020001 ): 0.059 KiB RPC/DCSInfo
DQMHistoSizes: changed ( 2024.0020001 ): 0.020 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0030001 ): 27.582 KiB PixelPhase1/Tracks
DQMHistoSizes: changed ( 2024.0030001 ): 6.617 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0030001 ): 1.564 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 2024.0030001 ): 0.164 KiB RPC/DCSInfo
DQMHistoSizes: changed ( 2024.0030001 ): -0.090 KiB JetMET/SUSYDQM
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

AdrianoDee · 2025-09-30T12:29:20Z

7e73ab20d08f4b7daeed642eee78ce1f  pdmv_group  cms:/store/data/Run2023B/ZeroBias/RAW/v1/000/366/403/00000/a68ba598-9216-4e0a-81dd-e9b3f2a6a7df.root                                                                                    REPLICATING[0/1/0]      T2_CH_CERN        1         4.031 GB                     2025-09-30 12:26:14
b36619eea268434d8e3cb2d6b63d155e  pdmv_group  cms:/store/data/Run2022B/ZeroBias/RAW/v1/000/355/094/00000/d47379ec-8b37-491d-9772-c320dc85eb09.root                                                                                    REPLICATING[0/1/0]      T2_CH_CERN        1         27.066 MB                    2025-09-30 12:25:04
3c967bf8a1604219bb89ac0d4878ae52  pdmv_group  cms:/store/data/Run2022C/JetHT/RAW/v1/000/355/862/00000/e712efc3-ba98-484a-91c0-c567e2c8b336.root                                                                                       REPLICATING[0/1/0]      T2_CH_CERN        1         1.676 GB                     2025-09-30 12:25:49

AdrianoDee · 2025-10-01T08:58:53Z

please test

cmsbuild · 2025-10-01T11:26:34Z

-1

Failed Tests: RelVals-INPUT
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35e700/48393/summary.html
COMMIT: dda7573
CMSSW: CMSSW_16_0_X_2025-09-30-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49020/48393/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

2022.00000012022.0000001_RunZeroBias2022B_10k/step2_RunZeroBias2022B_10k.log

Comparison Summary

The workflows 2025.0010001, 2025.0000001, 2024.0070001, 2024.0050001, 2024.0040001, 2024.0030001, 2024.0020001, 2024.0010001, 2024.0000001, 2023.0020001 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

You potentially added 1584 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 28125 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3504568
DQMHistoTests: Total failures: 40275
DQMHistoTests: Total nulls: 114
DQMHistoTests: Total successes: 3464159
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 77.366 KiB( 44 files compared)
DQMHistoSizes: changed ( 2024.0020001 ): 2.363 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0020001 ): 2.320 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 2024.0020001 ): 0.059 KiB RPC/DCSInfo
DQMHistoSizes: changed ( 2024.0020001 ): 0.020 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0030001,... ): 27.582 KiB PixelPhase1/Tracks
DQMHistoSizes: changed ( 2024.0030001 ): 6.617 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0030001 ): 1.564 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 2024.0030001 ): 0.164 KiB RPC/DCSInfo
DQMHistoSizes: changed ( 2024.0030001 ): -0.090 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0050001 ): 7.090 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0050001 ): ...
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

AdrianoDee · 2025-10-01T22:28:23Z

please test
(should be finally all staged)

gabrielmscampos · 2025-10-02T08:48:40Z

The golden JSONs are also available through CVMFS: /cvmfs/cms-griddata.cern.ch/cat/metadata/DC/Collisions<year>/latest

If the bot has access to it, this could be a good retry option in case the EOS connection fails.

AdrianoDee · 2025-10-02T11:51:50Z

The golden JSONs are also available through CVMFS: /cvmfs/cms-griddata.cern.ch/cat/metadata/DC/Collisions<year>/latest

If the bot has access to it, this could be a good retry option in case the EOS connection fails.

Right! Nice, I think it does (@smuzaffar?). I would leave this fix to go in and then push the extra check on cvmfs.

AdrianoDee · 2025-10-02T11:53:02Z

Also: are the tests stuck?

cmsbuild · 2025-10-02T13:02:16Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35e700/48413/summary.html
COMMIT: dda7573
CMSSW: CMSSW_16_0_X_2025-10-01-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49020/48413/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 2025.0010001, 2025.0000001, 2024.0070001, 2024.0050001, 2024.0040001, 2024.0030001, 2024.0020001, 2024.0010001, 2024.0000001, 2023.0020001 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

You potentially added 1582 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 28119 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3504568
DQMHistoTests: Total failures: 40299
DQMHistoTests: Total nulls: 114
DQMHistoTests: Total successes: 3464135
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 77.366 KiB( 44 files compared)
DQMHistoSizes: changed ( 2024.0020001 ): 2.363 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0020001 ): 2.320 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 2024.0020001 ): 0.059 KiB RPC/DCSInfo
DQMHistoSizes: changed ( 2024.0020001 ): 0.020 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0030001,... ): 27.582 KiB PixelPhase1/Tracks
DQMHistoSizes: changed ( 2024.0030001 ): 6.617 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0030001 ): 1.564 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 2024.0030001 ): 0.164 KiB RPC/DCSInfo
DQMHistoSizes: changed ( 2024.0030001 ): -0.090 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0050001 ): 7.090 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 2024.0050001 ): ...
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

AdrianoDee · 2025-10-02T18:00:24Z

+pdmv

cmsbuild · 2025-10-02T18:48:41Z

REMINDER @sextonkennedy, @mandrenguyen, @ftenchini: This PR was tested with #48999, please check if they should be merged together

AdrianoDee · 2025-10-02T20:00:18Z

hold
(may be superseded by #49051)

cmsbuild · 2025-10-02T20:00:37Z

Pull request has been put on hold by @AdrianoDee
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

AdrianoDee · 2025-10-03T05:53:20Z

Closed in favor of #49051

Not using golden when running tests in jenkins

dda7573

cmsbuild added this to the CMSSW_16_0_X milestone Sep 29, 2025

cmsbuild added pending-signatures tests-pending orp-pending pdmv-pending upgrade-pending code-checks-pending labels Sep 29, 2025

AdrianoDee mentioned this pull request Sep 29, 2025

Fixed Number of Events Data Wfs for Testing #48490

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 29, 2025

AdrianoDee marked this pull request as ready for review September 30, 2025 09:38

cmsbuild added tests-started and removed tests-pending labels Sep 30, 2025

cmsbuild added tests-rejected and removed tests-started labels Sep 30, 2025

cmsbuild added tests-started and removed tests-rejected labels Oct 1, 2025

cmsbuild added tests-rejected and removed tests-started labels Oct 1, 2025

cmsbuild mentioned this pull request Oct 1, 2025

Minor improvements to das-up-to-nevents.py #49045

Merged

cmsbuild added tests-started and removed tests-rejected labels Oct 1, 2025

This was referenced Oct 1, 2025

Bugfixes to TICL-barrel #48565

Merged

align HepMC3 plugin with HepMC2 plugin and add test configs #48999

Merged

cmsbuild added tests-approved and removed tests-started labels Oct 2, 2025

cmsbuild added pdmv-approved and removed pdmv-pending labels Oct 2, 2025

AdrianoDee mentioned this pull request Oct 2, 2025

Reading Certification JSON from cvmfs in das-up-to-nevents.py #49051

Merged

cmsbuild added the hold label Oct 2, 2025

AdrianoDee closed this Oct 3, 2025

AdrianoDee mentioned this pull request Dec 16, 2025

Removing JENKINS_PREFIX Check from das-up-to-nevents.py #49636

Merged

Conversation

AdrianoDee commented Sep 29, 2025

PR description:

PR validation:

Uh oh!

cmsbuild commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Sep 29, 2025

Uh oh!

AdrianoDee commented Sep 30, 2025

Uh oh!

cmsbuild commented Sep 30, 2025

Uh oh!

cmsbuild commented Sep 30, 2025

RelVals-INPUT

Comparison Summary

Uh oh!

AdrianoDee commented Sep 30, 2025

Uh oh!

AdrianoDee commented Oct 1, 2025

Uh oh!

cmsbuild commented Oct 1, 2025

RelVals-INPUT

Comparison Summary

Uh oh!

AdrianoDee commented Oct 1, 2025

Uh oh!

gabrielmscampos commented Oct 2, 2025

Uh oh!

AdrianoDee commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdrianoDee commented Oct 2, 2025

Uh oh!

cmsbuild commented Oct 2, 2025

Comparison Summary

Uh oh!

AdrianoDee commented Oct 2, 2025

Uh oh!

cmsbuild commented Oct 2, 2025

Uh oh!

AdrianoDee commented Oct 2, 2025

Uh oh!

cmsbuild commented Oct 2, 2025

Uh oh!

AdrianoDee commented Oct 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cmsbuild commented Sep 29, 2025 •

edited

Loading

AdrianoDee commented Oct 2, 2025 •

edited

Loading