[DONOTMERGE] docs/leios-design/LeiosFetchMarch2026Checkpoint.md - #848
[DONOTMERGE] docs/leios-design/LeiosFetchMarch2026Checkpoint.md#848nfrisby wants to merge 2 commits into
Conversation
|
|
||
| Even so, this prior art is an appropriate place to anchor the LeiosFetch implementation's design. | ||
| It describes four ideas that could be leveraged by the LeiosFetch logic. | ||
| - _Hedge requests_ are intentionally redundant requests, sent because---even among a centralized fleet---it's unknown which peer will be first to successfully reply. |
There was a problem hiding this comment.
Mention that praos does hedge requests for blocks.
| - **Challenge**: peers are permissionless. | ||
| Peers are identifiable only by their current IP address and port. | ||
| Some peers can be sampled from the stake pools' registration data, but not all peers---even upstream peers---are acquired that way. | ||
| (TODO is that fact a hard constraint on the design?) |
There was a problem hiding this comment.
No, it isn't a hard constraint on the design.
The ability to use unregistered peers as upstream peers was a requirement for on the network layer. It enables nodes that would other wise have been leaf nodes to fully participate. These nodes help the network by propagating blocks and TXs.
Unregistered nodes also provides a layer of protection for SPOs. An attacker may target their publicly registered relays but their blocks can still propagate through their unregistered relays.
| - **Challenge**: unpredictable latency and bandwidth, per peer. | ||
| A peer's past performance does not necessarily correlate with its future performance. | ||
| Even for completely trustworthy peers, the particular traffic route is dynamic and not necessarily reliable, since Cardano operates over the public Internet. | ||
| Moreover, not at all peers are trustworthy, so even if the physical performance hasn't changed, any peer could suddenly choose to respond much more slowly or not at all. | ||
| Timeouts limit the affect of so-called _low-and-slow_ attacks, but two principal difficulties remain. | ||
| - Timeout false alarms are harmful to the network topology and hence also delay EB diffusion, so the thresholds cannot be arbitrarily tight. | ||
| - It's not obvious what the tolerable performance lower bounds are nor how they should vary across time and/or peers. | ||
| - **Constraint**: LeiosFetch must quickly react to changes in peer behavior. |
There was a problem hiding this comment.
A peer's past performance does not necessarily correlate with its future performance.
Even for completely trustworthy peers, the particular traffic route is dynamic and not necessarily reliable, since Cardano operates over the public Internet.
That's why the
- we measure
Gusing keep-alive messages - we use a degenerate distribution (as a crude approximation) with exponential moving average when combining new measurements
We cannot predict the future (e.g. changes in global traffic routing, etc), but if a connection is regularly disturbed, then it will be reflected in our
| - **Challenge**: intercontinental peers are important. | ||
| Despite intercontinental peers inherently exhibiting poor latency, they drastically improve the diffusion of EBs through the global Cardano network when well-utilized. | ||
| Thus, There is no one-dimensional metric that would be sufficient for peer performance: a variety of peers is desired. | ||
| - This makes timeouts even more complicated to enforce. | ||
| - An intercontinental peer is best utilized by saturating its high Bandwidth-Delay Product (BDP), which requires proportional memory. | ||
| - **Constraint**: LeiosFetch should treat peers heterogenously (e.g. different BDPs). |
There was a problem hiding this comment.
For Praos we didn't have to make timeouts tight. Is there a reason why they should be tight for Leios?
It might be challenging to know BDP without disturbing the node. Can we assume/require minimum bandwidth (the delay we already measure using
| The Leios node has a limited amount of CPU, RAM, and local networking resources. | ||
| Popular nodes will serve > 100 downstream peers at once, so the server-side logic's must utilize resources frugally. | ||
| LeiosFetch must prevent adversarial peers---who cannot be identified a priori---from consuming too much more than their fair share of those resources. | ||
| - **Constraint**: LeiosFetch cannot simply serve every EB it offers. |
There was a problem hiding this comment.
An honest peer won't be able to consume every EB we offer at once, so the we don't need to serve every EB we offer instantly. If an honest client is struggling to process blocks it will request EBs in a slower way (there is some lag introduced by already pipelined requests, so pipelining depth should be chosen with care). It seems we just need to avoid serving more than a node could handle (e.g. some byte size limit on number of bytes in-flight per connection might be enough).
| - **Challenge**: EBs are large enough that their resource usage does not trivially scale. | ||
| CIP-0164 bounds the size of an EB at ≤ 12.512 MB; just 80 maximal EBs would already exceed 1 GB. | ||
| For example, if a popular node were to relay one maximal EB to every downstream peer, that alone would consume > 1.2512 GB of egress bandwidth. | ||
| - **Constraint**: LeiosFetch cannot simply request every EB from every peer that offers it. |
There was a problem hiding this comment.
We might want to support multiple NICs, this will not solve the challenge but it should allow larger egress bandwidth in more advanced setups.
| Shadow requests seem the least likely mechanism for LeiosFetch to adopt, since Cardano nodes typically simply disconnect when a peer performs below tolerances (e.g. times out). | ||
| Perhaps an intermediate probation stage could be useful, though. |
There was a problem hiding this comment.
If any error was thrown (including a timeout), we're reconnecting after a fixed daRepromoteDelay (10s). The peer might not end up in hot peers immediately - if it had a good score, it will be promoted in the next churn round. We might want a quicker way to get that peer to the hot set if it's score was descent, and slowly degrade it over time if it's current performance is not good enough.
| There, the concrete suggestion was two-fold. | ||
| - Upstream peers communicate with each other in order to cancel requests, since that way the cancellation can be sent as soon as the first response is _sent_ rather than _received_. | ||
| - Clients should send hedge requests only after a delay of "two times the average network message delay (1ms or less in modern data-center networks)". | ||
| That way, 95% of the time, the first server to succeed can send a cancellation before the other servers receive their copy of the client request. |
There was a problem hiding this comment.
We measure round trip with KeepAlive messages, which can be used for the hedge request delay.
| - Upstream peers communicate with each other in order to cancel requests, since that way the cancellation can be sent as soon as the first response is _sent_ rather than _received_. | ||
| - Clients should send hedge requests only after a delay of "two times the average network message delay (1ms or less in modern data-center networks)". | ||
| That way, 95% of the time, the first server to succeed can send a cancellation before the other servers receive their copy of the client request. | ||
| The recommendation of the average ping obviously conflicts with at least the "unpredictable latency and bandwidth" and "intercontinental peers are important" challenges. |
There was a problem hiding this comment.
Since all intercontinental peers will get this penalty, this will not influence upstreamyness nor fetchyness metrics used by churn. But there's a conflict, peers have an incentive to send their data as soon as possible to get a better score and remain hot at the other end (such connections disseminate own blocks).
| - Sending the cancellation as well in that case might avoid wasted upstream load. | ||
| That's also good, but secondary. | ||
|
|
||
| - In practice, however, an EB will always be offered by one peer (likely/intended to be an intercontinental peer for intercontinental EBs) before any other peer offers it. |
There was a problem hiding this comment.
This is true for intercontinental EBs, but once they reach a continent (or a particular data center), soon there will be multiple peers offering them simultaneously in that location if there are only handful intercontinental connections from that location.
|
Superseded by #1005 |
4 participants
I'm using this PR to foster discussion of design ideas with the Networking Team.