Fold in leios testnet bugfixes from 20260708 stuck sync incident - #2112
Draft
nfrisby wants to merge 5 commits into
Draft
Fold in leios testnet bugfixes from 20260708 stuck sync incident#2112nfrisby wants to merge 5 commits into
nfrisby wants to merge 5 commits into
IOG Hydra / ci/hydra-build:x86_64-linux.native.haskell96.checks.ouroboros-consensus:test:consensus-test
failed
Jul 9, 2026 in 55m 15s
Build failed
1 failed steps
Details
Failed Steps
Step 1
Derivation
/nix/store/9krbx4smq66lrxqrxa2vgz8ysrkdjxia-ouroboros-consensus-test-consensus-test-3.0.1.0-check.drv
Log
Running phase: unpackPhase
unpacking source archive /nix/store/mkj9c840klmi1kvw7v18gq68bk94whp4-source-test-consensus-test
source root is source-test-consensus-test
Running phase: patchPhase
Running phase: buildPhase
/nix/store/qib5z0qnprrmc38nacw6sq6ywhklzbmh-ouroboros-consensus-test-consensus-test-3.0.1.0/bin:
patching script interpreter paths in /build/tmp.K9QCacb9SG/bin
ouroboros-consensus
WallClock
delayNextSlot: OK (2.85s)
+++ OK, passed 10 tests.
delayClockShift: OK (1.16s)
+++ OK, passed 100 tests.
schedule goes back (100 in total):
90% True
10% False
schedule length (100 in total):
85% R_Gt 20
9% R_Btwn (10,20)
3% R_Eq 1
2% R_Btwn (5,10)
1% R_Btwn (4,5)
schedule skips (100 in total):
41% R_Btwn (10,20)
22% R_Gt 20
19% R_Btwn (5,10)
5% R_Eq 3
4% R_Btwn (4,5)
4% R_Eq 2
3% R_Eq 0
2% R_Eq 4
delayNoClockShift: OK (0.40s)
+++ OK, passed 1 test.
weighted Fait-Accompli committee selection tests
weighted Fait-Accompli committee selection model property tests
prop_stakeDistrDecreasingStakes: OK (1.43s)
+++ OK, passed 10000 tests.
prop_weightedFaitAccompliPersistentSeats_persistentSeatsRespectNumSeats: OK (4.03s)
+++ OK, passed 10000 tests.
Persistent to non-persistent seat ratio (10000 in total):
22.77% 90%-100%
16.51% 80%-90%
14.08% 50%-60%
13.56% 60%-70%
12.31% 70%-80%
7.22% 0%-10%
5.79% 40%-50%
4.39% 30%-40%
2.78% 20%-30%
0.59% 10%-20%
Stake distribution size (10000 in total):
10.00% 10-19
10.00% 20-29
10.00% 30-39
10.00% 40-49
10.00% 50-59
10.00% 60-69
10.00% 70-79
10.00% 80-89
10.00% 90-99
9.00% 0-9
1.00% 100-109
Target number of seats (10000 in total):
28.89% 0-9
20.57% 10-19
14.22% 20-29
10.81% 30-39
8.30% 40-49
6.77% 50-59
4.68% 60-69
2.97% 70-79
2.01% 80-89
0.76% 90-99
0.02% 100-109
prop_weightedFaitAccompliPersistentSeats_persistentSeatsHaveLargeStake: OK (4.61s)
+++ OK, passed 10000 tests; 2600 discarded.
More target seats than nodes (10000 in total):
100.00% False
Persistent to non-persistent seat ratio (10000 in total):
22.04% 90%-100%
17.45% 50%-60%
17.13% 80%-90%
15.15% 60%-70%
13.47% 70%-80%
6.44% 40%-50%
4.68% 30%-40%
2.96% 20%-30%
0.68% 10%-20%
Stake distribution size (10000 in total):
10.00% 10-19
10.00% 20-29
10.00% 30-39
10.00% 40-49
10.00% 50-59
10.00% 60-69
10.00% 70-79
10.00% 80-89
10.00% 90-99
9.00% 0-9
1.00% 100-109
Target number of seats (10000 in total):
26.21% 0-9
21.59% 10-19
15.44% 20-29
11.16% 30-39
8.44% 40-49
6.72% 50-59
4.71% 60-69
3.12% 70-79
1.84% 80-89
0.75% 90-99
0.02% 100-109
prop_weightedFaitAccompliPersistentSeats_totalStakePreserved: OK (4.93s)
+++ OK, passed 10000 tests.
More target seats than nodes (10000 in total):
95.57% False
4.43% True
Persistent to non-persistent seat ratio (10000 in total):
22.30% 90%-100%
16.12% 80%-90%
14.50% 50%-60%
13.44% 60%-70%
12.67% 70%-80%
6.94% 0%-10%
5.77% 40%-50%
4.75% 30%-40%
2.85% 20%-30%
0.66% 10%-20%
Stake distribution size (10000 in total):
10.00% 10-19
10.00% 20-29
10.00% 30-39
10.00% 40-49
10.00% 50-59
10.00% 60-69
10.00% 70-79
10.00% 80-89
10.00% 90-99
9.00% 0-9
1.00% 100-109
Target number of seats (10000 in total):
28.96% 0-9
20.09% 10-19
15.08% 20-29
10.56% 30-39
8.72% 40-49
6.46% 50-59
4.58% 60-69
2.79% 70-79
1.96% 80-89
0.75% 90-99
0.05% 100-109
prop_weightedFaitAccompliPersistentSeats_nonPersistentSeatsCountCorrect: OK (4.17s)
+++ OK, passed 10000 tests.
More target seats than nodes (10000 in total):
96.16% False
3.84% True
Persistent to non-persistent seat ratio (10000 in total):
23.13% 90%-100%
15.74% 80%-90%
13.90% 60%-70%
13.86% 50%-60%
12.57% 70%-80%
7.18% 0%-10%
5.65% 40%-50%
4.52% 30%-40%
2.85% 20%-30%
0.60% 10%-20%
Stake distribution size (10000 in total):
10.00% 10-19
10.00% 20-29
10.00% 30-39
10.00% 40-49
10.00% 50-59
10.00% 60-69
10.00% 70-79
10.00% 80-89
10.00% 90-99
9.00% 0-9
1.00% 100-109
Target number of seats (10000 in total):
29.32% 0-9
19.28% 10-19
14.65% 20-29
11.08% 30-39
8.54% 40-49
6.70% 50-59
4.51% 60-69
3.00% 70-79
2.21% 80-89
0.70% 90-99
0.01% 100-109
prop_weightedFaitAccompliPersistentSeats_residualSeatsDoNotQualify: OK (5.07s)
+++ OK, passed 10000 tests.
More target seats than nodes (10000 in total):
95.74% False
4.26% True
Persistent to non-persistent seat ratio (10000 in total):
22.69% 90%-100%
15.82% 80%-90%
14.30% 50%-60%
13.96% 60%-70%
12.13% 70%-80%
7.51% 0%-10%
5.62% 40%-50%
4.65% 30%-40%
2.74% 20%-30%
0.58% 10%-20%
Stake distribution size (10000 in total):
10.00% 10-19
10.00% 20-29
10.00% 30-39
10.00% 40-49
10.00% 50-59
10.00% 60-69
10.00% 70-79
10.00% 80-89
10.00% 90-99
9.00% 0-9
1.00% 100-109
Target number of seats (10000 in total):
29.42% 0-9
19.86% 10-19
14.42% 20-29
11.38% 30-39
8.34% 40-49
6.32% 50-59
4.35% 60-69
3.35% 70-79
1.89% 80-89
0.63% 90-99
0.04% 100-109
model conforms to Rust implementation: OK (9.30s)
+++ OK, passed 1000 tests.
HeaderValidation
validateIfCheckpoint
non-checkpoints are ignored: OK (0.02s)
+++ OK, passed 100 tests.
checkpoint matches should be accepted: OK (0.02s)
+++ OK, passed 100 tests.
checkpoint mismatches should be rejected: OK (0.02s)
+++ OK, passed 100 tests.
BlockFetchClient
blockFetch: OK (14.63s)
+++ OK, passed 100 tests.
Narrow CSJ
CaughtUp: OK (0.18s)
+++ OK, passed 100 tests.
ChainSyncClient
chainSync: OK (0.71s)
+++ OK, passed 100 tests:
77% NoMoreIntersection
13% ForkTooDeep
TickArrivalTimeStats (100 in total):
22% OnlyNotEarly_SomeEarly One Many
18% OnlyNotEarly_SomeEarly Zero Many
15% OnlyNotEarly_SomeEarly One Zero
13% OnlyNotEarly_SomeEarly Zero Zero
10% OnlyNotEarly_SomeEarly Many Zero
9% OnlyNotEarly_SomeEarly One One
7% OnlyNotEarly_SomeEarly Many One
6% OnlyNotEarly_SomeEarly Many Many
LocalStateQueryServer
localStateQueryServer: OK (0.86s)
+++ OK, passed 100 tests.
Acquired (7382 in total):
54.21% Success
41.59% AcquireFailurePointNotOnChain
4.20% AcquireFailurePointTooOld
Mempool
Mempool
TxSeq
lookupByTicketNo complete: OK (0.03s)
+++ OK, passed 100 tests.
lookupByTicketNo sound: OK
+++ OK, passed 100 tests:
62% successful miss
38% successful hit
splitAfterTxSize: OK
+++ OK, passed 100 tests.
splitAfterTxSizeSpec: OK (0.02s)
+++ OK, passed 100 tests.
IOSim properties
snapshotTxs == snapshotTxsAfter zeroTicketNo: OK (0.72s)
+++ OK, passed 100 tests:
95% non-empty Mempool
87% MempoolCapacityBytesOverride
13% NoMempoolCapacityBytesOverride
5% empty Mempool
valid added txs == getTxs: OK (2.87s)
+++ OK, passed 100 tests:
95% non-empty Mempool
89% MempoolCapacityBytesOverride
11% NoMempoolCapacityBytesOverride
5% empty Mempool
addTxs [..] == forM [..] addTxs: OK (3.94s)
+++ OK, passed 100 tests:
93% non-empty Mempool
89% MempoolCapacityBytesOverride
11% NoMempoolCapacityBytesOverride
7% empty Mempool
result of addTxs: OK (3.02s)
+++ OK, passed 100 tests:
96% non-empty Mempool
91% MempoolCapacityBytesOverride
9% NoMempoolCapacityBytesOverride
4% empty Mempool
Invalid transactions are never added: OK (2.94s)
+++ OK, passed 100 tests:
97% non-empty Mempool
90% MempoolCapacityBytesOverride
10% NoMempoolCapacityBytesOverride
3% empty Mempool
removeTxs: OK (0.92s)
+++ OK, passed 100 tests:
100% non-empty Mempool
86% MempoolCapacityBytesOverride
14% NoMempoolCapacityBytesOverride
0% empty Mempool
removeTxs [..] == forM [..] removeTxs: OK (1.72s)
+++ OK, passed 100 tests:
100% non-empty Mempool
88% MempoolCapacityBytesOverride
12% NoMempoolCapacityBytesOverride
0% empty Mempool
result of getCapacity: OK (0.97s)
+++ OK, passed 100 tests:
100% MempoolCapacityBytesOverride
94% non-empty Mempool
6% empty Mempool
0% NoMempoolCapacityBytesOverride
Added valid transactions are traced: OK (3.12s)
+++ OK, passed 100 tests:
92% non-empty Mempool
90% MempoolCapacityBytesOverride
10% NoMempoolCapacityBytesOverride
8% empty Mempool
Rejected invalid txs are traced: OK (3.17s)
+++ OK, passed 100 tests:
94% non-empty Mempool
90% MempoolCapacityBytesOverride
10% NoMempoolCapacityBytesOverride
6% empty Mempool
Removed invalid txs are traced: OK (0.63s)
+++ OK, passed 100 tests:
90% Removed some transactions
90% non-empty Mempool
83% MempoolCapacityBytesOverride
17% NoMempoolCapacityBytesOverride
10% empty Mempool
idx consistency: OK (1.96s)
+++ OK, passed 100 tests:
100% MempoolCapacityBytesOverride
100% empty Mempool
0% Mempool became empty
0% NoMempoolCapacityBytesOverride
0% The last transaction in the mempool is removed
0% non-empty Mempool
Mempool timeout: OK (2.43s)
+++ OK, passed 100 tests:
93% non-empty Mempool
87% MempoolCapacityBytesOverride
13% NoMempoolCapacityBytesOverride
7% empty Mempool
4% soft-timeout for otherwise-valid tx
addTextTx expectation (501 in total):
36.5% MtcAccepted
25.5% MtcRejected
22.2% MtcDiscard
13.4% MtcDisconnect
2.4% MtcNoSpace
Mempool fairness
There is no substantial bias in added transaction sizes: OK (4.51s)
QSM
sequential: OK (60.37s)
+++ OK, passed 1000 tests.
Command sequence length (1000 in total):
32.1% [0-10)
19.8% [10-20)
15.0% [20-30)
10.2% [30-40)
7.5% [40-50)
5.7% [50-60)
3.8% [60-70)
3.6% [70-80)
2.2% [80-90)
0.1% [90-100)
Commands (2908 in total):
31.91% TryAddTxs
22.97% GetSnapshot
22.56% ChangeLedger
22.56% SyncLedger
Maximum ticket number (1000 in total):
7.6% [0-5)
1.4% [40-45)
1.2% [15-20)
1.1% [10-15)
1.1% [110-115)
1.1% [25-30)
0.9% [100-105)
0.9% [160-165)
0.9% [30-35)
0.9% [85-90)
0.8% [105-110)
0.8% [115-120)
0.8% [180-185)
0.8% [220-225)
0.8% [255-260)
0.8% [380-385)
0.8% [50-55)
0.8% [595-600)
0.8% [675-680)
0.8% [70-75)
0.8% [80-85)
0.7% [140-145)
0.7% [150-155)
0.7% [190-195)
0.7% [235-240)
0.7% [245-250)
0.7% [265-270)
0.7% [275-280)
0.7% [285-290)
0.7% [320-325)
0.7% [35-40)
0.7% [355-360)
0.7% [370-375)
0.7% [410-415)
0.7% [440-445)
0.7% [45-50)
0.7% [470-475)
0.7% [60-65)
0.7% [95-100)
0.6% [130-135)
0.6% [165-170)
0.6% [170-175)
0.6% [185-190)
0.6% [250-255)
0.6% [260-265)
0.6% [290-295)
0.6% [345-350)
0.6% [375-380)
0.6% [55-60)
0.6% [650-655)
0.5% [125-130)
0.5% [175-180)
0.5% [20-25)
0.5% [210-215)
0.5% [230-235)
0.5% [305-310)
0.5% [315-320)
0.5% [330-335)
0.5% [360-365)
0.5% [450-455)
0.5% [485-490)
0.5% [510-515)
0.5% [535-540)
0.5% [640-645)
0.5% [715-720)
0.5% [75-80)
0.5% [775-780)
0.4% [1175-1180)
0.4% [1265-1270)
0.4% [135-140)
0.4% [145-150)
0.4% [195-200)
0.4% [240-245)
0.4% [270-275)
0.4% [295-300)
0.4% [325-330)
0.4% [340-345)
0.4% [350-355)
0.4% [385-390)
0.4% [405-410)
0.4% [425-430)
0.4% [460-465)
0.4% [475-480)
0.4% [5-10)
0.4% [500-505)
0.4% [520-525)
0.4% [530-535)
0.4% [540-545)
0.4% [550-555)
0.4% [580-585)
0.4% [590-595)
0.4% [610-615)
0.4% [625-630)
0.4% [65-70)
0.4% [655-660)
0.4% [695-700)
0.4% [705-710)
0.4% [735-740)
0.4% [800-805)
0.4% [925-930)
0.4% [935-940)
0.3% [1020-1025)
0.3% [1030-1035)
0.3% [1035-1040)
0.3% [1055-1060)
0.3% [1130-1135)
0.3% [1135-1140)
0.3% [1185-1190)
0.3% [120-125)
0.3% [1310-1315)
0.3% [1330-1335)
0.3% [1530-1535)
0.3% [155-160)
0.3% [200-205)
0.3% [205-210)
0.3% [215-220)
0.3% [225-230)
0.3% [310-315)
0.3% [365-370)
0.3% [395-400)
0.3% [400-405)
0.3% [415-420)
0.3% [455-460)
0.3% [490-495)
0.3% [495-500)
0.3% [555-560)
0.3% [560-565)
0.3% [615-620)
0.3% [630-635)
0.3% [660-665)
0.3% [665-670)
0.3% [720-725)
0.3% [725-730)
0.3% [745-750)
0.3% [755-760)
0.3% [785-790)
0.3% [825-830)
0.3% [850-855)
0.3% [855-860)
0.3% [860-865)
0.3% [885-890)
0.3% [90-95)
0.3% [905-910)
0.3% [910-915)
0.3% [940-945)
0.3% [945-950)
0.3% [950-955)
0.3% [960-965)
0.3% [970-975)
0.3% [975-980)
0.2% [1000-1005)
0.2% [1015-1020)
0.2% [1025-1030)
0.2% [1040-1045)
0.2% [1070-1075)
0.2% [1095-1100)
0.2% [1180-1185)
0.2% [1190-1195)
0.2% [1210-1215)
0.2% [1220-1225)
0.2% [1230-1235)
0.2% [1235-1240)
0.2% [1260-1265)
0.2% [1315-1320)
0.2% [1320-1325)
0.2% [1430-1435)
0.2% [1455-1460)
0.2% [1475-1480)
0.2% [1545-1550)
0.2% [1560-1565)
0.2% [1595-1600)
0.2% [1625-1630)
0.2% [1680-1685)
0.2% [280-285)
0.2% [420-425)
0.2% [430-435)
0.2% [435-440)
0.2% [465-470)
0.2% [480-485)
0.2% [505-510)
0.2% [515-520)
0.2% [525-530)
0.2% [545-550)
0.2% [575-580)
0.2% [585-590)
0.2% [600-605)
0.2% [670-675)
0.2% [690-695)
0.2% [700-705)
0.2% [760-765)
0.2% [770-775)
0.2% [780-785)
0.2% [820-825)
0.2% [830-835)
0.2% [845-850)
0.2% [880-885)
0.2% [900-905)
0.2% [990-995)
0.1% [1010-1015)
0.1% [1050-1055)
0.1% [1060-1065)
0.1% [1065-1070)
0.1% [1075-1080)
0.1% [1085-1090)
0.1% [1100-1105)
0.1% [1105-1110)
0.1% [1115-1120)
0.1% [1125-1130)
0.1% [1145-1150)
0.1% [1155-1160)
0.1% [1195-1200)
0.1% [1205-1210)
0.1% [1250-1255)
0.1% [1255-1260)
0.1% [1280-1285)
0.1% [1290-1295)
0.1% [1295-1300)
0.1% [1300-1305)
0.1% [1325-1330)
0.1% [1340-1345)
0.1% [1360-1365)
0.1% [1365-1370)
0.1% [1370-1375)
0.1% [1375-1380)
0.1% [1395-1400)
0.1% [1405-1410)
0.1% [1415-1420)
0.1% [1420-1425)
0.1% [1440-1445)
0.1% [1445-1450)
0.1% [1450-1455)
0.1% [1460-1465)
0.1% [1480-1485)
0.1% [1495-1500)
0.1% [1500-1505)
0.1% [1510-1515)
0.1% [1525-1530)
0.1% [1585-1590)
0.1% [1605-1610)
0.1% [1610-1615)
0.1% [1615-1620)
0.1% [1630-1635)
0.1% [1640-1645)
0.1% [1650-1655)
0.1% [1655-1660)
0.1% [1685-1690)
0.1% [1705-1710)
0.1% [1725-1730)
0.1% [1740-1745)
0.1% [1745-1750)
0.1% [1790-1795)
0.1% [1805-1810)
0.1% [1815-1820)
0.1% [1885-1890)
0.1% [2275-2280)
0.1% [300-305)
0.1% [335-340)
0.1% [390-395)
0.1% [445-450)
0.1% [565-570)
0.1% [605-610)
0.1% [635-640)
0.1% [645-650)
0.1% [680-685)
0.1% [710-715)
0.1% [730-735)
0.1% [750-755)
0.1% [765-770)
0.1% [795-800)
0.1% [805-810)
0.1% [810-815)
0.1% [840-845)
0.1% [870-875)
0.1% [875-880)
0.1% [895-900)
0.1% [915-920)
0.1% [920-925)
0.1% [930-935)
0.1% [955-960)
0.1% [965-970)
0.1% [980-985)
0.1% [985-990)
0.1% [995-1000)
Number of txs to add (18865 in total):
10.225% [60-70)
10.188% [70-80)
10.093% [80-90)
10.013% [30-40)
9.944% [10-20)
9.907% [40-50)
9.849% [90-100)
9.828% [20-30)
9.785% [50-60)
8.942% [0-10)
1.224% [100-110)
parallel
atomic: /nix/store/s3w5m3spa1g71hx0yb82lvk6394j3w5j-stdenv-linux/setup: line 1764: 32 Killed $drv/bin/consensus-test
33 Done | tee $out/test-stdout
Loading