Eager nodal reductions should respect MPI

[inducer]

cc @MTCam

[thomasgibson]

Have we sketched out what needs to be done to actually do this? Right now nodal reductions are rank-local. So could we perhaps introduce something similar to this, but for volumes?

Would this require extending DTAG_VOLUME_ALL to something like DTAG_VOLUME_PARTITION(rank_id) or something like that?

[inducer]

Would this require extending DTAG_VOLUME_ALL to something like DTAG_VOLUME_PARTITION(rank_id) or something like that?

I actually don't hate that. We definitely want to retain the ability to ask for rank-local reductions. One possible issue is that we probably also want to be able to ask for DTAG_VOLUME_PARTITION(rank=12) & DTAG_VOL_SCRAMJET_WALL. (before you panic: not arbitrary intersections, just one rank and one domain subset)

cc @majosm because this very much interacts with the subdomain stuff he is building.