Backport MPI-related developments [16.0.x]

[fwyzard]

PR description:

Backport of various MPI-related PRs to CMSSW 16.0.x:

• Make the group index deterministic when using MPI configuration splitter #50496
• Schedule MPI communication in runAsync to avoid deadlocks #50702
• Do not call MPI_Finalize on error #50752
• Extend edmMpiSplitConfig for multiple controllers #50678
• Reimplement MPI event synchronisation with async barriers #50810
• Add MPI modules to different paths in the splitter #50821

PR validation:

Tested in 17.0.x IBs, and in a dedicated high-statistic test based on CMSSW 16.1.x plus backports.

Backport status

Backport of various MPI-related PRs to CMSSW 16.0.x, to be tested online.

[fwyzard]

backport #50810

[fwyzard]

please test

[fwyzard]

+heterogeneous

[fwyzard]

type ngt

[cmsbuild]

A new Pull Request was created by @fwyzard for CMSSW_16_0_X.

It involves the following packages:

• HeterogeneousCore/MPICore (heterogeneous)
• HeterogeneousCore/MPIServices (heterogeneous)

@fwyzard, @makortel can you please review it and eventually sign? Thanks.
@makortel, @rovere this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

• Backported from Add MPI modules to different paths in the splitter #50821

[cmsbuild]

cms-bot internal usage

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_16_0_X IBs after it passes the integration tests and once validation in the development release cycle CMSSW_17_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

[cmsbuild]

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15ea6e/52895/summary.html
COMMIT: 8dd25a0
CMSSW: CMSSW_16_0_X_2026-04-26-0000/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50818/52895/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 2 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4172278
• DQMHistoTests: Total failures: 64
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4172194
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 197 edm output root files, 53 DQM output files
• TriggerResults: no differences found

[fwyzard]

backport #50821

[cmsbuild]

Pull request #50818 was updated. @cmsbuild, @fwyzard, @makortel can you please check and sign again.

[fwyzard]

please test

[fwyzard]

+heterogeneous

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_16_0_X IBs after it passes the integration tests and once validation in the development release cycle CMSSW_17_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

[cmsbuild]

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15ea6e/52916/summary.html
COMMIT: a7c81ac
CMSSW: CMSSW_16_0_X_2026-04-26-0000/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50818/52916/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4172278
• DQMHistoTests: Total failures: 67
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4172191
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 197 edm output root files, 53 DQM output files
• TriggerResults: no differences found

[ftenchini]

+1