[DRAFT] minimum changes to produce wf diffs seen in PR#47966 by mroguljic · Pull Request #48146 · cms-sw/cmssw

mroguljic · 2025-05-22T10:30:02Z

PR#47966 shows unexpected widely distributed small differences in test workflows. The minimal changes in this draft produce the same number of differences on a single workflow, 140.045, as the PR. This draft is created to allow the cms bot to run tests to check if the same happens in other workflows.

cmsbuild · 2025-05-22T10:30:26Z

cms-bot internal usage

cmsbuild · 2025-05-22T10:31:42Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48146/44916

There are other open Pull requests which might conflict with changes you have proposed:
- File CondFormats/SiPixelTransient/interface/SiPixelTemplate.h modified in PR(s): Adapt pixel cpe algo to better handle broken clusters #47966
- File CondFormats/SiPixelTransient/src/SiPixelTemplate.cc modified in PR(s): Adapt pixel cpe algo to better handle broken clusters #47966

mmusich · 2025-05-22T11:21:45Z

@cmsbuild, please test

fwyzard · 2025-05-22T11:29:40Z

allow @mroguljic test rights

cmsbuild · 2025-05-22T13:41:04Z

+1

Size: This PR adds an extra 44KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-32fe96/46322/summary.html
COMMIT: a031f18
CMSSW: CMSSW_15_1_X_2025-05-21-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48146/46322/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 25 lines from the logs
Reco comparison results: 4101 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4038193
DQMHistoTests: Total failures: 16754
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4021419
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

mroguljic · 2025-05-28T07:20:16Z

Per cmsbot, this accounts for roughly half the differences observed in the main PR. I've ran valgrind on one event in step 3 of 140.045 in two environments: reference (with no changes to cmssw) and reference + this PR. The two outputs can be found on lxplus at these locations: reference, PR

perrotta · 2025-06-20T07:20:53Z

please test
(Since the old ones already disappeared)

cmsbuild · 2025-06-20T10:26:24Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-32fe96/46847/summary.html
COMMIT: a031f18
CMSSW: CMSSW_15_1_X_2025-06-19-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48146/46847/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 24 lines from the logs
Reco comparison results: 3718 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4058157
DQMHistoTests: Total failures: 17094
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4041043
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-06-20T14:44:29Z

This PR contains too many commits (355 >= 240) and will not be processed.
Please ensure you have selected the correct target branch and consider squashing unnecessary commits.
The processing of this PR will resume once the commit count drops below the limit.

cmsbuild · 2025-07-07T08:40:59Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48146/45419

There are other open Pull requests which might conflict with changes you have proposed:
- File RecoLocalTracker/SiPixelRecHits/src/SiPixelTemplateReco.cc modified in PR(s): Reduce LogError noise when using (E)EoR3 CPE conditions #48263

cmsbuild · 2025-07-07T08:41:22Z

Pull request #48146 was updated.

mroguljic · 2025-07-07T08:41:54Z

please test

mroguljic · 2025-07-07T08:42:55Z

^ testing to see if declaration of variables at initialization will stabilize the output (as suggested here)

cmsbuild · 2025-07-07T10:54:51Z

+1

Size: This PR adds an extra 32KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-32fe96/47101/summary.html
COMMIT: f09e69d
CMSSW: CMSSW_15_1_X_2025-07-06-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48146/47101/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 6 lines to the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4067201
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4067170
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-07-07T11:04:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48146/45424

There are other open Pull requests which might conflict with changes you have proposed:
- File CondFormats/SiPixelTransient/src/SiPixelTemplate.cc modified in PR(s): Reduce LogError noise when using (E)EoR3 CPE conditions #48263
- File RecoLocalTracker/SiPixelRecHits/src/SiPixelTemplateReco.cc modified in PR(s): Reduce LogError noise when using (E)EoR3 CPE conditions #48263

mroguljic · 2025-07-07T11:04:54Z

please test

cmsbuild · 2025-07-07T11:05:06Z

Pull request #48146 was updated.

mroguljic · 2025-07-07T11:05:33Z

^ testing with assert to see if this has stabilized the output

cmsbuild · 2025-07-07T13:20:03Z

+1

Size: This PR adds an extra 56KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-32fe96/47109/summary.html
COMMIT: 4cb357c
CMSSW: CMSSW_15_1_X_2025-07-07-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48146/47109/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 22 lines to the logs
Reco comparison results: 3510 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4067201
DQMHistoTests: Total failures: 16886
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4050295
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

mmusich · 2025-07-08T10:37:03Z

^ testing with assert to see if this has stabilized the output

it does not appear to be the case, judging from #48146 (comment).

cmsbuild · 2025-09-10T06:55:56Z

Milestone for this pull request has been moved to CMSSW_16_0_X. Please open a backport if it should also go in to CMSSW_15_1_X.

cmsbuild · 2025-12-18T12:59:19Z

Milestone for this pull request has been moved to CMSSW_16_1_X. Please open a backport if it should also go in to CMSSW_16_0_X.

perrotta · 2026-02-25T09:44:51Z

@mroguljic do you still need this PR for your tests? Could you please close it, if not?
In the meanwhile, I'll give it a "-1" to keep it out of our radars

perrotta · 2026-02-25T09:44:55Z

-1

cmsbuild added this to the CMSSW_15_1_X milestone May 22, 2025

cmsbuild added reconstruction-pending db-pending pending-signatures tests-pending orp-pending code-checks-pending trk labels May 22, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels May 22, 2025

mroguljic mentioned this pull request May 22, 2025

Adapt pixel cpe algo to better handle broken clusters #47966

Closed

cmsbuild added tests-started and removed tests-pending labels May 22, 2025

cmsbuild added the allow-mroguljic label May 22, 2025

cmsbuild added tests-approved and removed tests-started labels May 22, 2025

cmsbuild mentioned this pull request Jun 18, 2025

Adapt template pixel cpe algo to better handle shortened or broken clusters #48356

Merged

cmsbuild added tests-started and removed tests-approved labels Jun 20, 2025

cmsbuild added tests-approved and removed tests-started labels Jun 20, 2025

cmsbuild added tests-pending code-checks-pending and removed db-pending tests-approved code-checks-approved labels Jul 7, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 7, 2025

cmsbuild added tests-started and removed tests-pending labels Jul 7, 2025

cmsbuild added tests-approved and removed tests-started labels Jul 7, 2025

now with assert

4cb357c

cmsbuild removed the tests-approved label Jul 7, 2025

mmusich mentioned this pull request Jul 8, 2025

Reduce LogError noise when using (E)EoR3 CPE conditions #48263

Merged

mroguljic mentioned this pull request Jul 8, 2025

Numerical instabilities related to SiPixelTemplate.cc #48499

Open

Conversation

mroguljic commented May 22, 2025

Uh oh!

cmsbuild commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented May 22, 2025

Uh oh!

mmusich commented May 22, 2025

Uh oh!

fwyzard commented May 22, 2025

Uh oh!

cmsbuild commented May 22, 2025

Comparison Summary

Uh oh!

mroguljic commented May 28, 2025

Uh oh!

perrotta commented Jun 20, 2025

Uh oh!

cmsbuild commented Jun 20, 2025

Comparison Summary

Uh oh!

cmsbuild commented Jun 20, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

mroguljic commented Jul 7, 2025

Uh oh!

mroguljic commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Comparison Summary

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

mroguljic commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Uh oh!

mroguljic commented Jul 7, 2025

Uh oh!

cmsbuild commented Jul 7, 2025

Comparison Summary

Uh oh!

mmusich commented Jul 8, 2025

Uh oh!

cmsbuild commented Sep 10, 2025

Uh oh!

cmsbuild commented Dec 18, 2025

Uh oh!

perrotta commented Feb 25, 2026

Uh oh!

perrotta commented Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cmsbuild commented May 22, 2025 •

edited

Loading