Add true edge-granular redistribution (balance-edges-strict) - #99
Open
niklas-uhl wants to merge 26 commits into
Open
Add true edge-granular redistribution (balance-edges-strict)#99niklas-uhl wants to merge 26 commits into
niklas-uhl wants to merge 26 commits into
Conversation
Member
|
Apart from the failing unit test, Codex's review: This one might be worth investigating Probably depends on how you plan to use Probably can ignore this one, since |
DanielSeemaier
approved these changes
Jul 15, 2026
Collaborator
Author
|
Addressed the Codex findings: Strict balancing computed before dedup — confirmed real.
New/extended regression tests cover exact balance under cross-PE duplicates, complete-partition coverage including isolated vertices, and the |
niklas-uhl
force-pushed
the
feature/true-edge-balance-redistribution
branch
from
August 20, 2026 09:31
bbda169 to
ef17ca6
Compare
ComputeBalancedEdgeDistribution's breakpoint loop previously emitted a breakpoint for every bucket boundary a single vertex's degree crossed, pushing the same vertex value multiple times. Distribution then handed several consecutive PEs an empty (v, v) range, starving them entirely even when other PEs' worth of edges/vertices were available. Cap this at one breakpoint per vertex, attributing the vertex causing the overflow to the *current* bucket (absorbing the overflow) and starting the next bucket at the following vertex. Since the pushed value is now always strictly greater than the previous one, it can never collide with an already-used breakpoint -- which also fixes a related, pre-existing collision when the overflowing vertex is the very first vertex of a PE's range (its deferred breakpoint could coincide with that PE's own start, e.g. rank 0's hardcoded initial 0). Add a regression test with a dominant hub vertex plus a uniform overlay of edges among the rest, asserting no PE is starved to an empty range.
Adds the new GraphRedistribution/GraphDistribution enum value and its "balance-edges-strict" CLI/option-string key, plumbed through the map and operator<< helpers. Option-string parsing (context.cpp) already applies "redistribution" generically to every generator, so this also becomes reachable via GenerateFromOptionString without further changes. No generator handles the new value yet -- the two switches over these enums (graph500_generator.cpp, io.cpp) now warn via -Wswitch until they're wired up in follow-up commits, which is the intended transitional state (mirrors the plan's suggested build-first-verify approach; querying it now would silently produce an empty graph).
Adds a new postprocessor primitive that splits a single vertex's own
edges across multiple PEs when necessary, achieving exact (+/-1)
edge-count balance even in the presence of a hub vertex whose degree
alone exceeds one PE's fair share -- something the existing
vertex-atomic RedistributeEdgesBalanced cannot do by design.
Algorithm: reuses the existing "lightweight metadata exchange, then a
single edge-data shuffle" shape of ComputeBalancedEdgeDistribution,
extended so the metadata phase (routing (vertex, degree) pairs to
each vertex's nominal owner by vertex ID, not by where its edges
currently happen to live) also resolves, for every vertex, the exact
global edge rank of each contributing source PE's first local edge of
that vertex. Each source PE then locally derives every one of its
edges' final target PE from that one base rank plus its own offset,
against a distribution array every PE can compute independently --
no second edge-data shuffle. A vertex is correctly split across
however many adjacent PEs its degree demands (not capped at 2), since
the split point is resolved analytically rather than assumed to fall
within a locally-sorted concatenation of edges.
(An earlier, simpler design -- local sort per PE plus MPI_Exscan of
raw local edge counts -- was considered, but does not actually
guarantee correct splitting when a vertex's edges are scattered
across many non-adjacent PEs before redistribution, which is the
normal case for a hub vertex in RMAT before this call.)
Adds Graph::has_split_vertices and Generator::{Set,}HasSplitVertices
to carry the "did any vertex actually get split" signal through
Finalize()/Take() for the compatibility guard added in a later commit.
Extends the existing parameterized redistribute_edges_test.cpp fixture
with RedistributeEdgesTrueBalance as a third case (wrapped down to a
VertexRange via .fully_owned_vertex_range) for the edge-multiset
tests; the ownership-invariant tests are skipped for it since true
splitting intentionally breaks single-PE ownership for boundary
vertices. Adds a dedicated adversarial test with a dominant hub vertex
asserting exact (+/-1) balance and that the hub's edges land on more
than one PE once there is more than one PE to split it across.
Graph500Generator::FinalizeEdgeList (shared by RMAT and Kronecker) now handles GraphRedistribution::BALANCE_EDGES_TRUE via RedistributeEdgesTrueBalance, propagating has_split_vertices. Adds a --redistribution CLI option (factored into a shared add_option_redistribution helper) to gnm-directed, gnm-undirected, rgg2d, rgg3d, and kronecker as well, ahead of wiring their generators in follow-up commits. While smoke-testing RMAT with balance-edges-strict, found that duplicate edges could survive redistribution: a generator like RMAT symmetrizes every directed sample unconditionally and samples independently per chunk, so it can legitimately draw the same undirected edge twice on two different source PEs (particularly likely for hub-adjacent pairs, i.e. exactly the vertices this feature splits). The vertex-atomic modes always catch such duplicates because routing is a pure function of vertex ID, so both copies are always co-located and caught by a final per-PE dedup. True-split routing is per-edge (based on position within a vertex's aggregated edge list), so two duplicate copies of a split vertex's edge could legitimately be routed to different target PEs and never meet. Fix: after determining a vertex is split (its degree spans more than one PE's fair edge share), escalate its actual edges -- not just a count -- to the vertex's owner instead of routing them directly. The owner gathers every contributing sender's data (bounded by the split vertices' own total degree, not the whole graph, since the vast majority of vertices aren't split and keep using the cheap direct routing), deduplicates by head value, and forwards each surviving edge straight to its true target. This also means any duplicate that does survive dedup at this stage is guaranteed to straddle at most one boundary between two adjacent PEs (equal values are adjacent once sorted), matching the invariant the vertex-atomic modes get for free. Adds a regression test modeling the extreme case (every PE contributes an identical copy of the same hub) confirming the duplicates collapse to exactly the expected unique edge count with no duplicates surviving in the gathered output.
Adds the new mode to the existing parameterized distribution/density matrix (skipping OwnershipInvariant, which assumes single-PE vertex ownership) and to the cross-distribution edge-set-identity check against balance-vertices. The latter specifically exercises real RMAT generation through the public GenerateFromOptionString API rather than a synthetic edge list, which would have caught the split-vertex dedup gap fixed in the previous commit.
Adds a local_edges_ staging buffer to both GNMDirected and GNMUndirected<BigInt>, redirecting their PushEdge call sites into it, plus a FinalizeEdgeList override handling all three GraphRedistribution modes. balance-vertices is a true no-op: GNM's chunked generation already confines every edge's tail to its own PE's [start_node_, end_node_) range by construction (see the local_row / to >= start_node_ guards in GNMUndirected, and the recursive chunk-confined sampling in GNMDirected), so redistributing would just reproduce the same placement at the cost of an unnecessary full shuffle -- this must not regress the default setting's performance. balance-edges and balance-edges-strict route through RedistributeEdgesBalanced / RedistributeEdgesTrueBalance as usual, with remap_round_robin=false (GNM has no ID-correlated skew the way RMAT does, so there's nothing to gain from scrambling vertex IDs). Adds --redistribution to the gnm-directed/gnm-undirected CLI subcommands (via the add_option_redistribution helper introduced for RMAT/Kronecker) and a new gnm_distribution_test.cpp mirroring the RMAT distribution test structure across both directed/undirected variants and all three redistribution modes.
Adds FinalizeEdgeList overrides to the concrete RGG2D/RGG3D classes
(the shared Geometric2D/Geometric3D::GenerateEdgeList() is final and
also shared with Delaunay, out of scope, so it can't live there).
balance-vertices is a no-op, matching the same reasoning applied to
GNM: Geometric{2,3}D::GenerateEdgeList() already confines every
edge's tail to this PE's own chunk-assigned vertex_range, which also
encodes spatial locality worth preserving -- redistributing would
just reproduce the same placement at the cost of an unnecessary full
shuffle, regressing the default setting's performance.
Adds two guards, both throwing ConfigurationError for edge-balanced
redistribution (balance-edges/-strict) combined with:
- --coordinates: splitting/reshuffling vertices would require a
second coordinate-shuffling exchange primitive not built here.
- --edgeweights-generator=euclidean-distance: RGG pushes these
weights directly into graph_.edge_weights during generation, in
lockstep with graph_.edges; the redistribution primitives only
shuffle Edgelist, so redistributing afterwards would desync
weights from edges.
Widens the try/catch in GenerateInMemory (in_memory_facade.cpp) to
cover the whole generate/finalize/weight pipeline, not just
NormalizeParameters -- these new guards fire from inside Finalize(),
and previously that would have produced an uncaught abort instead of
the intended clear error message.
Fixes an unrelated pre-existing bug found while testing the
euclidean-distance guard: GetEdgeWeightGeneratorTypeMap() mapped the
string "euclidean_distance" to HASHING_BASED instead of
EUCLIDEAN_DISTANCE, making that generator unreachable via the CLI/
option-string API.
Adds --redistribution to rgg2d/rgg3d (already done for the CLI in an
earlier commit) and a new rgg_distribution_test.cpp: the usual
parameterized ownership/duplicate/cross-distribution matrix, plus a
dedicated negative-test suite driving the low-level Factory/Generator
API directly (GenerateFromOptionString funnels through
GenerateInMemory, which now catches and aborts on ConfigurationError,
so it can't be used to observe the exception in gtest).
Removes the "not implemented" stub for REQUIRES_REDISTRIBUTION readers (plain edgelist etc.) combined with BALANCE_EDGES: they now read an arbitrary partition and defer to FinalizeGraphFragment's postprocessing pass, the same way BALANCE_VERTICES already does for them. For BALANCE_EDGES_TRUE, every reader -- including METIS/ParHIP, which are FindNodeByEdge-capable and normally bypass postprocessing entirely via an efficient offset-based range read -- now reads an ordinary vertex-balanced partition (a plain vertex-range read, which every reader supports) and lets FinalizeGraphFragment's postprocessing pass do the real redistribution via RedistributeEdgesTrueBalance. This intentionally does NOT throw for METIS/ParHIP: FindNodeByEdge only supports the vertex-atomic approximation BALANCE_EDGES needs, not the exact positions true balancing requires, but a plain vertex-range read works for any reader. FinalizeGraphFragment now takes InputGraphConfig so it can dispatch on config.distribution instead of hardcoding vertex balance for every REQUIRES_REDISTRIBUTION reader regardless of what was actually requested (ROOT/EXPLICIT/BALANCE_VERTICES keep that pre-existing behavior, matching the plan's explicit scope; BALANCE_EDGES and BALANCE_EDGES_TRUE get their own real dispatch). Found and fixed two additional bugs while getting this to actually work end-to-end: - None of the redistribution primitives carry vertex_weights/ edge_weights alongside the Edgelist they redistribute, so redistributing a weighted graph would silently desync weights from their edges/vertices -- a pre-existing gap for BALANCE_EDGES, not just the new BALANCE_EDGES_TRUE path. Added a guard in FinalizeGraphFragment that throws ConfigurationError instead (collectively -- a PE with an empty local partition would otherwise see empty weight arrays locally even though the graph as a whole is weighted, causing some PEs to throw while others proceed into collective redistribution code, deadlocking). - generic_file_generator_test.cpp's ReadStaticGraphOnRoot helper called Finalize(MPI_COMM_WORLD) from inside an `if (rank == 0)` block, on a generator constructed as if it were the only PE (rank=0, size=1). This was harmless while Finalize() was a no-op for METIS/ParHIP, but deadlocks now that BALANCE_EDGES_TRUE performs real collective communication there; fixed to use MPI_COMM_SELF, matching the single-PE semantics the generator was already constructed with. Relocates ConfigurationError from kagen/generators/generator.h to kagen/definitions.h (generator.h now includes it, so existing consumers are unaffected) so kagen/io.cpp can throw it without a layering inversion. Extends generic_file_generator_test.cpp: adds BALANCE_EDGES_TRUE to the existing (format, distribution, representation) parameterization for METIS/ParHIP (EDGE_LIST only for BALANCE_EDGES_TRUE -- CSR requires single vertex ownership, which a split-vertex graph violates, the same restriction adjacency-grouped output formats have; this test fixture also bypasses the compatibility guard that will reject that combination at the CLI/library entry points, added in a follow-up commit), and updates the weighted-graph tests to expect ConfigurationError for BALANCE_EDGES_TRUE.
Adds a centralized check in GenerateInMemory, right after Finalize() and before GenerateEdgeWeights/GenerateVertexWeights/statistics run: collectively reduces (MPI_LOR) each PE's Generator::HasSplitVertices() into a single any_split flag, then rejects combining an actual split with anything that assumes single-PE vertex ownership: - adjacency-grouped output formats (METIS, HMETIS variants, DOT variants, ParHIP, XtraPuLP, freight-netl variants, netd-are), - --validate-simple-graph, - a cross-PE-lookup edge-weight generator (hashing-based/ uniform-random; both do AllgatherVertexRange+FindPEInRange and would silently misbehave on split vertices), - statistics (guarded by the same !config.quiet condition the actual statistics computation uses further down, since --quiet skips it regardless of statistics_level). Only fires when a split *actually happened*, not merely when BALANCE_EDGES_TRUE was requested -- a graph with no real hub under that mode is fully compatible with everything. The decision logic lives in its own function, CheckSplitVertexCompatibility(any_split, config), rather than inlined in GenerateInMemory, specifically so it can be unit-tested directly: GenerateInMemory funnels through MPI_Abort on ConfigurationError (by design, so the CLI fails cleanly), which makes EXPECT_THROW-style testing impossible against the full pipeline. The new compatibility_guard_test.cpp exercises the decision matrix (each adjacency-grouped format, validate-simple-graph, both cross-PE-lookup edge-weight generators, statistics with/without --quiet, multiple output formats) without driving any MPI collectives. Adds a doc comment to ValidateVertexRanges (kagen/tools/validator.cpp) noting it hard-requires single-PE vertex ownership and is not adapted to tolerate split vertices, per the plan's explicit scope (no code change there -- the guard above already prevents reaching it).
Adds tests/kronecker/kronecker_distribution_test.cpp, mirroring the RMAT distribution test pattern (ownership/duplicate/cross-distribution checks across all three redistribution modes) -- Kronecker mechanically shares Graph500Generator::FinalizeEdgeList with RMAT, but wasn't covered by a dedicated test yet. Adds tests/file/plain_edgelist_distribution_test.cpp covering the one reader category generic_file_generator_test.cpp cannot: a REQUIRES_REDISTRIBUTION reader (plain edgelist), where balance-edges used to throw "not implemented" and balance-edges-strict needs the same "read an arbitrary partition, then postprocess" treatment as balance-edges now does. Uses a hub-and-ring graph so there's a real hub to exercise splitting under balance-edges-strict. Found and fixed a real deadlock while building this test: computing a unique-per-process temp filename via getpid() doesn't work when called from every rank, since each MPI rank is a separate OS process with its own PID -- only rank 0 ends up writing to the name it computed, while every other rank looks for a *different*, nonexistent file of its own and hangs trying to read it. Fixed by having only rank 0 compute the name (and write the file), then broadcasting it to the other ranks.
Adds a new "Distributing the Generated Graph Across PEs" section explaining the three modes (balance-vertices, balance-edges, balance-edges-strict), their tradeoffs, and balance-edges-strict's compatibility restrictions (edge-list-shaped consumption only; the additional --coordinates / euclidean-distance-edge-weights restriction for the geometric generators). Previously this was only documented via inline CLI --help text. Adds the --redistribution flag to the CLI usage blocks for gnm-directed/gnm-undirected, rgg2d/rgg3d, rmat, and kronecker, and updates the File Graph Generator's --distribution line to include balance-edges-strict.
Found via manual testing after the fact: KaGen::UseCSRRepresentation() combined with a graph that ends up with split vertices (from --redistribution=balance-edges-strict) segfaulted instead of hitting the compatibility guard. Root cause: the guard in GenerateInMemory runs *after* generator->Finalize() returns, but for the affected generators, the actual CSR-building step (BuildCSRFromEdgeList, which assumes every edge's tail lies within vertex_range -- untrue for a split vertex's edges) happens *inside* Finalize(): EdgeListOnlyGenerator::FinalizeCSR and FileGraphGenerator::FinalizeCSR both call FinalizeEdgeList() (which performs the redistribution and determines has_split_vertices) and then immediately build CSR from the result, all within the same call. By the time Finalize() returns (or, as it turned out, crashes), the out-of-bounds write already happened. The facade-level guard can only meaningfully react to CSR representation once has_split_vertices is already known, which is too late for this specific path. Fixed by checking graph_.has_split_vertices immediately after FinalizeEdgeList() in both FinalizeCSR implementations, before BuildCSRFromEdgeList() runs, throwing ConfigurationError instead. This is safe to do independently per generator (no additional Allreduce needed) since has_split_vertices is already collectively consistent by this point -- RedistributeEdgesTrueBalance Allreduces it before returning. Also extends CheckSplitVertexCompatibility (the facade-level guard) with a GraphRepresentation parameter and its own CSR check, as defense in depth for the general case (representation is known before generation starts, unlike has_split_vertices) and to document the restriction in one place; updates its call site and unit tests accordingly.
The GNU/Release matrix leg hung twice in a row in ctest (around test_plain_edgelist_distribution.4cores), stalling the job for the full 60-minute job timeout instead of failing fast. No test in the suite normally takes anywhere near 120s, so this only fires on a genuine hang and turns a 60-minute stall into a ~2-minute failure that names the specific hung test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RedistributeEdgesTrueBalance computed its edge-rank bucket boundaries from pre-deduplication degree counts, so cross-PE duplicate edges left holes once they were removed, breaking the promised +/-1 balance. A cheap rank-based rebalance pass now runs after dedup to close that drift without ever concentrating a hub's full edge set on one PE. Separately, fully_owned_vertex_range (assigned directly to Graph::vertex_range) silently excluded isolated (degree-0) vertices and one side of every split boundary vertex, corrupting header/global vertex counts and per-vertex-weight generation even when no split actually occurred. Boundary detection is now a single Allgather instead of a pairwise Sendrecv, producing a complete, gap-free partition of [0, n) as vertex_range, plus a restored, stricter fully_owned_vertex_range for consumers that need single-PE adjacency ownership. SplitVertexInfo/partial_vertices, previously computed internally and discarded at every call site, is now exposed on Graph so a split-aware downstream consumer can use it directly instead of re-deriving split detection itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Read each PE's strict edge slice straight from disk for --distribution=balance-edges-strict on eligible formats, avoiding the vertex-balanced-read + all-to-all RedistributeEdgesTrueBalance pass (2x memory + communication). Adds a GraphReader::ReadStrictEdgeRange capability: - ParHIP: native CSR (split-aware, partial boundary rows) or edge list, from a single contiguous adjncy slice via the offset header. - METIS: sequential-scan edge slice; native split CSR built locally via BuildCSRFromEdgeList over the row-space vertex_range. - weighted-binary-edgelist: edge-indexed slice, gated on a cheap global tail-sortedness check; unsorted input falls back to redistribution. Boundary/split metadata is computed from one MPI_Allgather (no edge movement); the tail-based computation is factored out of RedistributeEdgesTrueBalance into shared ComputeBoundaryOwnership / ComputeEdgeBalancedBoundaries helpers. Also extend --distribution=balance-edges to the direct path for sorted edge-list readers: read the strict slice, then heal partial boundary vertices to whole-vertex (vertex-atomic) ownership via a single weight-carrying neighbor exchange (HealToVertexAtomic), falling back to RedistributeEdgesBalanced when a single exchange cannot resolve ownership (empty PE or a vertex spanning >2 PEs) or the input is unsorted. Edge-weighted input now succeeds on the direct path (weights stay attached); vertex-weighted input is rejected (a split vertex's whole weight is ambiguous). The CSR + split-vertex guard is relaxed for the directly-read case; the corrupting build-from-edgelist CSR paths still throw earlier. CLI help text corrected. Tests: parameterized strict_edge_balance_test over ParHIP+METIS (edge list + split CSR); new weighted_edgelist_distribution_test (direct balance-edges/-strict with weight preservation + unsorted fallback); updated compatibility_guard and generic_file_generator expectations. Also disambiguate parhip_reader_writer temp filenames by comm size to fix a pre-existing concurrent-run flake. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previously a split graph (--redistribution=balance-edges-strict) could
only be produced as CSR by the direct strict ParHIP file read; converting
a split edge list to CSR threw ("CSR representation requires single-PE
vertex ownership"). That excluded all in-memory generators (rmat, gnm) and
edge-list file inputs from producing a truly edge-balanced CSR graph.
EdgeListOnlyGenerator::FinalizeCSR and FileGraphGenerator::FinalizeCSR now
build a split CSR by extending the row space down by one vertex on a
left-partial (replica) PE, whose first vertex is credited to the lower-rank
canonical PE and would otherwise underflow BuildCSRFromEdgeList's
`from - vertex_range.first`. The result is the same physically-present
row-space layout the ParHIP reader produces: adjacent PEs' ranges overlap
by one vertex at each split, described by partial_vertices. Trailing
isolated vertices are still absorbed as empty rows.
Also relax the statistics guard: basic statistics (vertex/edge counts and
imbalance) need only per-PE counts and stay valid for a split graph -- edge
imbalance in particular is exactly what balance-edges-strict targets -- so
only advanced statistics (per-vertex degree distribution, density,
locality, ghost nodes) are rejected now.
Document the CSR-vs-edge-list vertex_range distinction in kagen.h (the
gap-free contract holds for edge lists; CSR overlaps by one at splits).
Tests: cover strict-edge-balanced CSR generation for rmat and update the
compatibility guard for the basic-statistics carve-out. Fixes a latent
test bug where the rmat distribution tests passed `distribution=` in the
option string, which generators ignore (the key is `redistribution=`), so
they never actually exercised balance-edges/-strict.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous note claimed the CSR overlap only arises from a strict file read because edge-list-to-CSR conversion of a split graph throws -- but the prior commit removed that throw. Reframe around the actual invariant: the range is gap-free in every case except a split graph in CSR representation (both the file read and the edge-list-to-CSR conversion produce the overlapping row space), and note the summed-size consequence of the overlap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…R graphs vertex_range deliberately overlaps by one vertex at each split boundary in the CSR representation; these give the gap-free, single-owner range/count instead, matching what edge-list representation already provides. Replaces the ambiguous partial_vertices vector with explicit left_partial_vertex/right_partial_vertex fields on Graph and EdgeBalancedDistribution, since a PE whose entire local CSR is one shared row could not otherwise tell locally whether that vertex was shared with the lower- or higher-rank neighbor -- which silently dropped a vertex from the true count in that case. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vertex_range previously meant different things per representation: for EDGE_LIST it was the gap-free ownership range, but for a split CSR graph it was the overlapping physically-present row space instead. Any generic consumer of a Graph (e.g. EdgeRange::FromGraph, a library caller not aware of the CSR special case) that read vertex_range directly -- as its name suggests it should be safe to do -- would silently miscount or misindex a split graph. vertex_range is now always the gap-free, single-owner partition of [0, n), identically for both representations; Graph::PhysicalVertexRange() gives the (possibly-overlapping) set of vertices a PE holds any neighborhood information for, which CSR's xadj/adjncy happen to be indexed by. This replaces the TrueVertexRange()/TrueNumberOfLocalVertices() added earlier this session, which are now redundant since vertex_range already has that meaning. Fixes two latent bugs this uncovered: EdgeRange::FromGraph mapped CSR rows to the wrong global vertex ids on a split graph, and PrintBasicStatistics double-counted a split boundary vertex for CSR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-balance boundary ReadStrictEdgeRange locates a PE's physical row window via FindNodeByEdge, which finds vertices by the edges they own. A run of degree-0 vertices sitting exactly at a from_edge/to_edge boundary owns no edges, so it's invisible to that lookup and gets skipped from the physically read xadj window -- even though ComputeBoundaryOwnership's gap-free partition still credits those vertices to whichever PE precedes them. This left xadj one row short of PhysicalVertexRange() on the affected PE, misaligning every row from the boundary onward for CSR consumers indexing by global vertex id via PhysicalVertexRange(). Backfill degenerate (zero-length) rows for any such absorbed run so xadj stays aligned with PhysicalVertexRange(); a genuine split vertex already has its row physically present, so this never fires for those. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…alone repro script KaCCv2 hangs during kagen generation with redistribution=balance-edges-strict at large scale (768 ranks, n=201326592, m=1610612736). The breadcrumbs bracket every collective in the true-balance redistribution path so the last one printed without a matching "after ..." pinpoints where a run gets stuck. The script isolates the KaGen call (via the CLI options subcommand) from KaCCv2/MultiStep entirely, using the same SLURM/env setup as the job that hung. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The first repro run hung right at the stage-2 residual-rebalance MPI_Exscan/ MPI_Allreduce, with rank 0 having finished everything before it -- meaning some other rank never reached that collective, but root-only breadcrumbs can't say which one. Add KAGEN_TRUE_BALANCE_DEBUG_ALL_RANKS=1 to print from every rank (tagged with its own rank number) so the straggler can be found; default stays root-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… fixed The exclusive_scan int32-overflow fix (77572ee) was confirmed to resolve the hang at full scale, so the instrumentation added to find it is no longer needed. Keeps the fix itself, drops the breadcrumb prints and the KAGEN_TRUE_BALANCE_DEBUG_ALL_RANKS env toggle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same class of bug as 77572ee: exclusive_scan's accumulator type is deduced from the init value, not the output iterator, so a plain 0 (int) silently overflows once the running total exceeds INT32_MAX. - tests/gather.h: GatherCSR's xadj prefix sum (SInt) -- affects gathering a graph with >~2^31 edges. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
niklas-uhl
force-pushed
the
feature/true-edge-balance-redistribution
branch
from
August 20, 2026 16:09
2ac6109 to
d6b9728
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
balance-edgesbreakpoint computation (PEs could end up with zero edges for oversized/hub vertices).balance-edges-strict(BALANCE_EDGES_TRUE), which can split a single vertex's own edges across multiple PEs for exact edge-count balance, via a metadata-phase + single-shuffle primitive (RedistributeEdgesTrueBalance) with escalation-based dedup for split-vertex duplicate edges.--redistribution/--distributionoptions into RMAT, Kronecker, GNM (directed/undirected), RGG2D/3D, and file-graph reading (METIS/ParHIP/plain-edgelist), without adding overhead to any generator's default (vertex-balanced) path.CheckSplitVertexCompatibility) that throws a clearConfigurationErrorinstead of corrupting data or crashing when split vertices are combined with something that assumes single-PE vertex ownership: adjacency-grouped output formats (METIS/HMETIS/DOT/ParHIP/XtraPuLP/freight-netl/netd-are), CSR representation,--validate-simple-graph, statistics, and hashing/uniform-random edge weight generators. Also guards vertex/edge-weighted input against silent desync under any edge-balanced redistribution of file graphs.--edgeweights-generator=euclidean_distanceto the wrong generator type.Test plan
redistribute_edges_test,rmat_distribution_test,gnm_distribution_test,rgg_distribution_test,generic_file_generator_test,kronecker_distribution_test,plain_edgelist_distribution_test,compatibility_guard_test.ctest -j4suite: 61/63 passing; only the pre-existing, known-flakytest_parhip_reader_writerfails intermittently under parallel load (passes in isolation), confirmed unrelated to this branch.ConfigurationError.