Remove PortableHostMultiCollection and PortableDeviceMultiCollection by Electricks94 · Pull Request #49734 · cms-sw/cmssw

Electricks94 · 2026-01-07T09:12:33Z

This PR removes PortableHostMultiCollection and PortableDeviceMultiCollection and replaces them with SoABlocks (see #48629).

This has two major advantages:

Simplifications of dictionaries because variadic templates are not necessary anymore
Code simplifications because BlocksView holds views to all sublayouts

In a future PRs the code can be simplified even more because with kernels that use a BlocksView as input can work on the entire data. Hence, it is not necessarry anymore to have every sub-view as input. In this PR, however, I tried to make as few changes as possible to not blow up the PR too much.

@felicepantaleo @leobeltra fyi

cmsbuild · 2026-01-07T09:13:08Z

cms-bot internal usage

Electricks94 · 2026-01-07T09:14:30Z

type ngt

cmsbuild · 2026-01-07T09:15:34Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49734/47318

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Portable/README.md modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233
- File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072, Enabling usage of filters in MPI offloading #49233
- File DataFormats/Portable/scripts/portableHostCollectionHints modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824
- File DataFormats/PortableTestObjects/src/classes_def.xml modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824
- File DataFormats/SoATemplate/interface/SoACommon.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072
- File DataFormats/TrackingRecHitSoA/interface/TrackingRecHitsHost.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File RecoLocalTracker/Phase2TrackerRecHits/plugins/alpaka/Phase2OTRecHitsSoAConverter.cc modified in PR(s): [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346
- File RecoTracker/LSTCore/src/classes_def.xml modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824
- File RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtuplet.cc modified in PR(s): Use TkPixelCPERecord for PixelCPEFastParams* #46852, [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346
- File RecoTracker/PixelTrackFitting/plugins/PixelTrackProducerFromSoAAlpaka.cc modified in PR(s): [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346

cmsbuild · 2026-01-07T09:15:57Z

A new Pull Request was created by @Electricks94 for master.

It involves the following packages:

DQM/SiPixelHeterogeneous (dqm)
DataFormats/HGCalDigi (simulation)
DataFormats/Portable (heterogeneous)
DataFormats/PortableTestObjects (heterogeneous)
DataFormats/SoATemplate (heterogeneous)
DataFormats/TrackSoA (heterogeneous, reconstruction)
DataFormats/TrackingRecHitSoA (heterogeneous, reconstruction)
DataFormats/VertexSoA (heterogeneous, reconstruction)
HeterogeneousCore/AlpakaTest (heterogeneous)
RecoLocalTracker/Phase2TrackerRecHits (reconstruction)
RecoLocalTracker/SiPixelRecHits (reconstruction)
RecoTauTag/HLTProducers (hlt)
RecoTracker/LST (reconstruction)
RecoTracker/LSTCore (reconstruction)
RecoTracker/PixelSeeding (reconstruction)
RecoTracker/PixelTrackFitting (reconstruction)
RecoVertex/PixelVertexFinding (reconstruction)

@Martin-Grunewald, @Moanwar, @civanch, @cmsbuild, @ctarricone, @fwyzard, @gabrielmscampos, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @mmusich, @nothingface0, @rseidita, @srimanob can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @IzaakWN, @VinInn, @VourMa, @azotz, @dgulhan, @dkotlins, @elusian, @fabiocos, @felicepantaleo, @ferencek, @fioriNTU, @gpetruc, @idebruyn, @jandrea, @makortel, @martinamalberti, @mbluj, @missirol, @mmasciov, @mmusich, @mroguljic, @mtosi, @pfs, @rovere, @threus, @tsusa this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2026-01-07T09:50:23Z

enable gpu

mmusich · 2026-01-07T09:50:37Z

test parameters:

workflows = ph2_hlt

fwyzard · 2026-01-07T09:52:28Z

-```
-The layouts should not be added as parameters for the device collection. Those script can be use equally with the
-single layout collections or multi layout collections.
+Multi layout collections have been removed and replaced by `SoABlocks`. See the `DataFormats/SoATemplate`


Seems broken off ?

fwyzard · 2026-01-07T09:55:12Z


 collectionName = sys.argv[1]

+# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


I don't think this ever supported PortableMultiCollections.
Do we need a special syntax for the SoA blocks ?

For the SoA blocks no special syntax is needed. It is equivalent to a normal PortableCollection. I adapted the file to be consistent with #48824. Which means first Collection declaration. Then Collection::Layout followed by the SoALayout that is aliased by Collection::Layout. Lastly the definition with edm::Wrapper

Then, can you remove the TODO ?

fwyzard · 2026-01-07T09:56:55Z


 import sys
-
+# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


I think you should remove the part if len(layouts) > 1:

fwyzard · 2026-01-07T09:59:54Z


  using TestSoA3 = TestSoALayout3<>;

+  GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(firstLayout, TestSoALayout), SOA_BLOCK(secondLayout, TestSoALayout2))


maybe just

Suggested change

GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(firstLayout, TestSoALayout), SOA_BLOCK(secondLayout, TestSoALayout2))

GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(first, TestSoALayout), SOA_BLOCK(second, TestSoALayout2))

?

Same below.

fwyzard · 2026-01-07T10:02:42Z

+  // Either int32_t for normal layouts or std::array<int32_t, N> for SoABlocks layouts
+  auto size() const {
+    return layout_.metadata().size();
+    ;


Suggested change

;

fwyzard · 2026-01-07T10:08:15Z


 // generic SoA-based product in host memory
+// TODO: Should we define a size alias for the number of elements? Also should be switch to unsigned and/or 64 bit?
+// Exeeding the size of int32_t fails without clear error messages.


This needs to be an int32_t (actually a plain int) because of how ROOT stores the columns 🤷🏻‍♂️ .

I understand. How about allowing any integral type for the constructor of the PortableCollections and then implementing a check if the value can be savely casted to int? If it is save to cast we would do so and pass it on to the Layout, otherwise we throw an error.

I just realised that the PortableCollections have already a constructor that takes std::integral as input type for the elements. So the idea would be just to align all other constructors as well and add a check for the cast.

fwyzard · 2026-01-07T10:08:35Z

    alpaka::memset(std::forward<TQueue>(queue), *buffer_, 0x00);
  }

-  // part of the ROOT read streamer


Why remove the comment ?

fwyzard · 2026-01-07T10:09:38Z

+  // Either int32_t for normal layouts or std::array<int32_t, N> for SoABlocks layouts
+  auto size() const {
+    return layout_.metadata().size();
+    ;


Suggested change

;

cmsbuild · 2026-01-14T13:16:11Z

Pull request #49734 was updated. @Martin-Grunewald, @Moanwar, @civanch, @ctarricone, @fwyzard, @gabrielmscampos, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @mmusich, @nothingface0, @rseidita, @srimanob can you please check and sign again.

makortel

Could I ask you to double check the changes to the classes_def.xml files, after the rebase over #48824 ?

Done. I found one case that needs to be adjusted, the others looked fine by eye.

makortel · 2026-01-14T14:55:22Z

  <class name="reco::HitPortableCollectionHost"/>
-  <!-- needed to avoid auto parsing by ROOT. Order in the file matters -->
-  <class name="reco::HitPortableCollectionHost::Implementation"/>


An entry for

<class name="reco::HitPortableCollectionHost::Layout"/>

is needed, and it needs to be placed before entry of reco::TrackingBlocksLayout. I'd suggest to place it (maybe both of these lines) before all the Layout entries. I'd also suggest to include a comment that it needs to be placed before the aliased-to type (like in other files).

cmsbuild · 2026-01-14T15:09:13Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49734/47484

Found files with invalid states:
- HeterogeneousCore/AlpakaTest/test/testAlpaka.root:
  - Added: 5e9971d
  - Deleted: 57cee3f
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Portable/README.md modified in PR(s): Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233
- File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072, Enabling usage of filters in MPI offloading #49233
- File DataFormats/TrackingRecHitSoA/interface/TrackingRecHitsHost.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File RecoLocalTracker/Phase2TrackerRecHits/plugins/alpaka/Phase2OTRecHitsSoAConverter.cc modified in PR(s): [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346
- File RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtuplet.cc modified in PR(s): Use TkPixelCPERecord for PixelCPEFastParams* #46852, [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346
- File RecoTracker/PixelTrackFitting/plugins/PixelTrackProducerFromSoAAlpaka.cc modified in PR(s): [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346

cmsbuild · 2026-01-14T15:09:40Z

Pull request #49734 was updated. @Martin-Grunewald, @Moanwar, @civanch, @cmsbuild, @ctarricone, @fwyzard, @gabrielmscampos, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @mmusich, @nothingface0, @rseidita, @srimanob can you please check and sign again.

Electricks94 · 2026-01-14T15:28:39Z

please test

cmsbuild · 2026-01-14T20:11:17Z

-1

Failed Tests: RelVals RelVals-AMD_W7900
Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82823b/50625/summary.html
COMMIT: 5d8b297
CMSSW: CMSSW_16_1_X_2026-01-14-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49734/50625/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

34634.999

----- Begin Fatal Exception 14-Jan-2026 18:01:35 CET-----------------------
An exception of category 'OutOfBound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 0
   [1] Running path 'HLTriggerFinalPath'
   [2] Prefetching for module TriggerSummaryProducerAOD/'hltTriggerSummaryAOD'
   [3] Prefetching for module L1HPSPFTauProducer/'l1tHPSPFTauProducer'
   [4] Prefetching for module L1TPFCandMultiMerger/'l1tLayer1'
   [5] Prefetching for module L1TCorrelatorLayer1Producer/'l1tLayer1HGCal'
   [6] Calling method for module HGCalBackendLayer2Producer/'l1tHGCalBackEndLayer2Producer'
Exception Message:
TC X1 = 0.0713466 out of the seeding histogram bounds 0.076 - 0.58
----- End Fatal Exception -------------------------------------------------

Failed RelVals-AMD_W7900

34634.40334634.403_TTbar_14TeV+Run4D121PU_Patatrack_PixelOnlyAlpaka_Validation/step2_TTbar_14TeV+Run4D121PU_Patatrack_PixelOnlyAlpaka_Validation.log
34634.40234634.402_TTbar_14TeV+Run4D121PU_Patatrack_PixelOnlyAlpaka/step2_TTbar_14TeV+Run4D121PU_Patatrack_PixelOnlyAlpaka.log
34634.75134634.751_TTbar_14TeV+Run4D121PU_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D121PU_HLT75e33TimingAlpaka.log

Expand to see more relval errors ...

fwyzard · 2026-01-20T14:29:30Z

+heterogeneous

fwyzard · 2026-01-20T16:08:37Z

ignore tests-rejected with ib-failure

mmusich · 2026-01-20T16:16:45Z

+hlt

for the changes in RecoTauTag/HLTProducers/src/L2TauTagNNProducerAlpaka.cc

fwyzard · 2026-01-20T16:18:30Z

@Electricks94 can you open a backport for 16.0.x ?

jfernan2 · 2026-01-20T16:20:02Z

+1

kpedro88 · 2026-01-20T16:48:55Z

+simulation

gabrielmscampos · 2026-01-21T09:04:43Z

+dqm

cmsbuild · 2026-01-21T09:05:11Z

This pull request is fully signed and it will be integrated in one of the next master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2026-01-21T09:24:11Z

+1

makortel · 2026-01-21T21:35:12Z

This PR caused 4 DQM integration test to fail, see #49892

fwyzard · 2026-01-21T21:59:19Z

I guess it's to be expected if the DQM tests reuse old streamer files as input ?

As far as I know there is no guarantee that streamer files are readable with any release different from what was used to write them.

makortel · 2026-01-21T22:35:52Z

As far as I know there is no guarantee that streamer files are readable with any release different from what was used to write them.

Correct.

cmsbuild added this to the CMSSW_16_1_X milestone Jan 7, 2026

cmsbuild added reconstruction-pending simulation-pending dqm-pending hlt-pending pending-signatures tests-pending orp-pending code-checks-pending heterogeneous-pending tracking trk changes-dataformats labels Jan 7, 2026

cmsbuild added the ngt label Jan 7, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 7, 2026

fwyzard reviewed Jan 7, 2026

View reviewed changes

makortel reviewed Jan 14, 2026

View reviewed changes

Correct dictonary definition for TrackingRecHitSoA

5d8b297

Electricks94 mentioned this pull request Jan 19, 2026

Refactor of PortableCollections #49860

Merged

Electricks94 mentioned this pull request Jan 21, 2026

[Backport to 16_0_X (#49734)] Remove PortableHostMultiCollection and PortableDeviceMultiCollection #49882

Merged

cmsbuild mentioned this pull request Jan 21, 2026

Run the ONNX test on GPU only if CUDA is available #49883

Merged

makortel mentioned this pull request Jan 21, 2026

DictionaryNotFound exception in DQM/Integration unit tests #49892

Closed

smuzaffar mentioned this pull request Jan 21, 2026

RecoTrackerLSTCore-standalone-compilation test failed #49893

Closed

slava77 mentioned this pull request Jan 21, 2026

LST standalone fix after interface change in Portable*MultiCollection #49895

Merged

This was referenced Jan 22, 2026

update the streamer files for the DQMGPUvsCPU stream cms-data/DQM-Integration#17

Merged

update run number used for streamDQMGPUvsCPU based client tests #49898

Merged

cmsbuild mentioned this pull request Jan 23, 2026

Merging CMSSW_16_1 into CMSSW_16_1_EVOLUTION_X #49910

Merged

Electricks94 mentioned this pull request Jan 28, 2026

[Backport to 16_0_X (#49860)] Refactor of PortableCollections #49953

Merged

ariostas mentioned this pull request Feb 2, 2026

Fix typo in LST causing segfault #50001

Merged


		collectionName = sys.argv[1]

		# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


		import sys

		# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


		using TestSoA3 = TestSoALayout3<>;

		GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(firstLayout, TestSoALayout), SOA_BLOCK(secondLayout, TestSoALayout2))

Conversation

Electricks94 commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Electricks94 commented Jan 7, 2026

Uh oh!

cmsbuild commented Jan 7, 2026

Uh oh!

cmsbuild commented Jan 7, 2026

Uh oh!

mmusich commented Jan 7, 2026

Uh oh!

mmusich commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

makortel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

Electricks94 commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Failed RelVals

Failed RelVals-AMD_W7900

Uh oh!

fwyzard commented Jan 20, 2026

Uh oh!

fwyzard commented Jan 20, 2026

Uh oh!

mmusich commented Jan 20, 2026

Uh oh!

fwyzard commented Jan 20, 2026

Uh oh!

jfernan2 commented Jan 20, 2026

Uh oh!

kpedro88 commented Jan 20, 2026

Uh oh!

gabrielmscampos commented Jan 21, 2026

Uh oh!

cmsbuild commented Jan 21, 2026

Uh oh!

mandrenguyen commented Jan 21, 2026

Uh oh!

Electricks94 commented Jan 7, 2026 •

edited

Loading

cmsbuild commented Jan 7, 2026 •

edited

Loading