Skip to content

Fold in leios testnet bugfixes from 20260708 stuck sync incident - #2112

Draft
nfrisby wants to merge 5 commits into
leios-prototypefrom
nfrisby/fold-in-leios-testnet-bugfixes
Draft

Fold in leios testnet bugfixes from 20260708 stuck sync incident#2112
nfrisby wants to merge 5 commits into
leios-prototypefrom
nfrisby/fold-in-leios-testnet-bugfixes

Conversation

@nfrisby

@nfrisby nfrisby commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

This PR fixes three bugs.

  • We were accidentally using the default method headerContainsLeiosCert for the HFC, so we essentially never interepted MsgRollForward as an offer. I have no idea how the manual testing had done for that original PR didn't fail despite that.
  • The pruning logic for the set of EBs acquired was incorrect: it was assuming that all the announcements were necessarily in the VolDB. But---somewhat counterintuitively---our syncing node is downloading the EB before it even downloads the announcer's RB. So this pruning logic was instantly deleting the EB.
  • There was a deeply fundamental one-line bug in the implementation of the pipelined LeiosFetch client: if some unprocessed responses had arrived, we would sometimes (at the very last possible moment) discard a request instead of actually sending it to the upstream peer. But the LeiosFetch decision logic's accounting wasn't informed, so it eventually concluded too many requests were outstanding (even though they were never actually sent) for it to be able to send any more.

This code has been running on the testnet for 12 hrs now, without exhibiting the stuck syncs (which has been perfectly reliable prior to these).

nfrisby added 5 commits July 9, 2026 05:06
It's ultimate Set EbHash, except it can be GC'd according to the
SlotNo's that caused the EbHash to be in the set.
But the LeiosFetch logic didn't know that, so it eventually thought
too many bytes were inflight even though they weren't, so it stopped
sending any new requests.
@nfrisby nfrisby changed the title Nfrisby/fold in leios testnet bugfixes Fold in leios testnet bugfixes from 20260708 stuck sync incident Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant