diff --git a/docker-compose.yaml b/docker-compose.yaml index 53f4603e3..6a1cc3a82 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -261,6 +261,8 @@ services: - AR_IO_NODE_RELEASE=${AR_IO_NODE_RELEASE:-82-pre} - CHUNK_POST_MIN_SUCCESS_COUNT=${CHUNK_POST_MIN_SUCCESS_COUNT:-} - CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT=${CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT:-} + - CHUNK_POST_MIN_DISTINCT_DOMAINS=${CHUNK_POST_MIN_DISTINCT_DOMAINS:-} + - CHUNK_POST_PREFERRED_SOFT_FALLBACK=${CHUNK_POST_PREFERRED_SOFT_FALLBACK:-} - CHUNK_POST_MAX_CONSECUTIVE_FAILURES=${CHUNK_POST_MAX_CONSECUTIVE_FAILURES:-} - CHUNK_POST_SORTED_PEERS_CACHE_DURATION_MS=${CHUNK_POST_SORTED_PEERS_CACHE_DURATION_MS:-} - CHUNK_POST_RESPONSE_TIMEOUT_MS=${CHUNK_POST_RESPONSE_TIMEOUT_MS:-} diff --git a/docs/INDEX.md b/docs/INDEX.md index 0280530a1..e2c60810e 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -40,6 +40,7 @@ Fast, offline lookups for data item to root transaction mappings. | Document | Description | |----------|-------------| | [Optimistic Chunk Ingest Cache](chunk-ingest-cache.md) | Validating and caching chunks posted to the gateway, with confirmation-driven cleanup | +| [Chunk Fan-out Seeding](chunk-fanout-seeding.md) | Fault-domain-aware chunk broadcast — operator setup, safety model (why it can't lose data), monitoring, and rollback | ## Data Export diff --git a/docs/chunk-fanout-seeding.md b/docs/chunk-fanout-seeding.md new file mode 100644 index 000000000..520ee7844 --- /dev/null +++ b/docs/chunk-fanout-seeding.md @@ -0,0 +1,189 @@ +# Chunk Fan-out Seeding (fault-domain aware) + +Operator guide for the fault-domain-aware chunk fan-out: what it does, how to +enable and manage it safely, what to watch, and — most importantly — **why it +cannot lose data**. + +> **TL;DR for operators:** Both flags default OFF, and with them off this release +> is byte-identical to the previous one (regression-tested + live-soaked). The +> flags only change *when a chunk POST is reported successful* and *how widely it +> fans out* — they never store or delete data. Turning them on is safe when your +> uploader re-verifies permanence (Turbo does). To roll back: set the flags to +> their defaults and recreate `core`. + +## 1. What this is + +When an uploader (the Turbo bundler) posts a chunk to this gateway +(`POST /chunk`), the gateway **fans it out** to multiple Arweave peers so the +chunk spreads through the network toward miners. This feature makes that fan-out +**fault-domain aware**: + +- Peers are bucketed by **fault domain** — IP `/24` (IPv4) / `/48` (IPv6), a + proxy for "independent operator / network block". +- The gateway can require/measure landings across **distinct** fault domains, + not just a raw count of `200`s. +- It fixes a **single-point-of-failure**: the five preferred "tip" nodes + (`tip-1..5.arweave.xyz`) all live in one `/24` (`38.29.227.0/24`), and the + default quorum hard-required 2 tip successes — so a tip outage would fail every + chunk POST even when many independent peers accepted the chunk. + +The gateway is a **seeding *target/relay***, not the durability owner. Durability +(the guarantee that data permanently lands on-chain) is owned by the **uploader**, +which re-verifies permanence across multiple gateways and re-seeds anything that +doesn't confirm. Keep that layering in mind — it is why this change is safe. + +## 2. Data safety — why this cannot lose data + +This is the important part. There are three independent reasons a chunk cannot be +lost by this change: + +1. **Default-off is a no-op.** With `CHUNK_POST_MIN_DISTINCT_DOMAINS=0` and + `CHUNK_POST_PREFERRED_SOFT_FALLBACK=false` (the defaults), the quorum and + fan-out are **byte-identical** to the previous release. A regression test pins + this, and it was verified in a live production soak. Doing nothing changes + nothing. + +2. **The gateway never holds the only copy.** `broadcastChunk` *relays* a chunk + to Arweave peers. It does not store the sole copy, and these flags do not touch + the gateway's local chunk cache (the separate, unchanged optimistic ingest + cache). Nothing here deletes or drops data. + +3. **The uploader owns durability and re-verifies.** A chunk-POST result is + *advisory to the uploader*, not the final word on durability. Turbo + independently checks permanence across multiple gateways and **re-seeds + (redrive → repack → re-post)** anything that doesn't reach permanence. So even + if a gateway *reported* success on peers that turned out not to propagate, the + uploader re-checks and re-seeds — no data is lost. + +### Per-flag safety + +- **`CHUNK_POST_MIN_DISTINCT_DOMAINS`** — cannot lose data in either direction. + It is **best-effort**: it never fails a POST (so no false-failures) and never + gates success (so no false-successes). It only drives fan-out breadth and emits + a metric. Worst case: extra outbound bandwidth. It also cannot cause + under-seeding — early termination only fires once the base quorum is already + met. + +- **`CHUNK_POST_PREFERRED_SOFT_FALLBACK`** — the only flag that changes *when a + POST is reported successful*. It lets a POST succeed on a strong distinct-domain + quorum of discovered peers when the tips fall short. This is safe **for a + re-verifying uploader** (Turbo): the gateway's `200` is advisory, and Turbo + confirms permanence independently and re-seeds if needed. Note the strict 2-tip + quorum was *also* never a durability guarantee (tips can fail to propagate too); + this flag only changes the *composition* of the reported quorum, not whether the + uploader re-verifies. + + > **Operational precondition (read this):** the fallback trades a stricter + > success signal for availability. It is safe when the uploader re-verifies + > permanence and re-seeds — which **Turbo does**. If this gateway fronts a + > *naive* uploader that treats a `200` as final and never re-checks, then in a + > pathological case (tips down **and** every discovered peer accepts-but-doesn't- + > propagate) a chunk could be reported seeded but not reach miners. For gateways + > serving arbitrary/naive uploaders, **leave this OFF**. For a Turbo-fronting + > gateway, it is safe to enable. + +## 3. The flags + +| Env var | Default | Effect | +|---|---|---| +| `CHUNK_POST_MIN_DISTINCT_DOMAINS` | `0` (off) | Best-effort diversity target (distinct `/24`·`/48` domains). Drives fan-out breadth + a `below_target` metric; **never fails a POST**. Rejects non-integer / negative values at startup. | +| `CHUNK_POST_PREFERRED_SOFT_FALLBACK` | `false` | When on, a POST meets quorum via a strong distinct-domain discovered quorum whenever the preferred (tip) quorum falls short — removing the tip-`/24` SPOF. **Softens the tip requirement in steady state too** (see §5). | + +Related existing knobs (unchanged): `CHUNK_POST_MIN_SUCCESS_COUNT` (3), +`CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT` (2), `CHUNK_POST_PEER_CONCURRENCY`, +`CHUNK_POST_ABORT_TIMEOUT_MS` (per-peer, default 2 s), +`CHUNK_POST_RESPONSE_TIMEOUT_MS` (per-peer, default 5 s). See `docs/envs.md`. + +> Config is read at startup from `.env`. Changing a flag requires recreating the +> `core` container for it to take effect. + +## 4. Recommended rollout (staged, safe) + +1. **Deploy with flags OFF.** Zero behavior change — but the metrics, the + `X-AR-IO-Chunk-Placement-Domains` response header, and the Grafana "Chunk + Fan-out Seeding" dashboard row **populate immediately**. Use this to learn your + baseline placement diversity (watch the `distinct_domains` p10 — a low p10 + means chunks are landing on few independent operators today). This step is + pure observability and carries no risk. +2. **Enable `CHUNK_POST_PREFERRED_SOFT_FALLBACK=true`** (recreate `core`). This + removes the tip-`/24` SPOF: a tip outage no longer fails chunk POSTs. Confirm + the precondition in §2 first (your uploader re-verifies — Turbo does). +3. **(Optional) Raise `CHUNK_POST_MIN_DISTINCT_DOMAINS`** (e.g. to `2` or `3`) to + push the fan-out toward more independent operators. This is best-effort (never + fails a POST) but **costs outbound bandwidth** — the fan-out keeps contacting + peers to chase the target. On *fresh* chunks it is largely bounded by the + preferred set anyway (discovered peers reject an unknown `data_root` until the + tx propagates); broader diversity accrues over the seeding **lifecycle** + (re-posts as the tx propagates), not on a single POST. + +## 5. Behavior change to understand (steady state) + +With `CHUNK_POST_PREFERRED_SOFT_FALLBACK=true`, the effective success quorum +becomes: + +> **`MIN_PREFERRED_SUCCESS_COUNT` tip successes OR `MIN_SUCCESS_COUNT` total +> successes across ≥ `max(MIN_PREFERRED_SUCCESS_COUNT, MIN_DISTINCT_DOMAINS)` +> distinct fault domains.** + +This applies **in steady state, not only during a full outage** — a normal POST +where only one tip acks but several independent discovered peers land will now +pass, where the strict path would have failed. This is the intended availability +win, but it means **the unconditional "2 tip successes" guarantee no longer holds +when the flag is on.** `chunk_post_preferred_shortfall_total` stays emitted so tip +shortfalls remain visible even when the fallback carries the POST. + +## 6. What to monitor + +Grafana ships a **"Chunk Fan-out Seeding"** row (auto-provisioned). Key signals: + +| Metric | What it tells you | +|---|---| +| `chunk_post_distinct_domains` (avg / p50 / **p10**) | How many independent operators each chunk reaches. p10 is the exposure tail. | +| `chunk_post_preferred_shortfall_total{reason}` | Tip health / when the fallback engages. `tips_unavailable` = tips ineligible; `tips_failed` = tips reachable but didn't ack. Emitted even when the fallback carries the POST — a rising rate is a tip-health signal, not a failure. | +| `chunk_post_domain_shortfall_total{reason=below_target}` | Advisory: POSTs landing on fewer domains than `MIN_DISTINCT_DOMAINS`. Never a failure. | +| `arweave_chunk_broadcast_total{status}` | Overall POST success/fail rate. This should **not drop** after enabling either flag (see §7). | + +## 7. Troubleshooting / unintended-behavior checks + +- **Chunk POST success rate drops after enabling a flag** → unexpected, investigate. + Neither flag should *reduce* success: `SOFT_FALLBACK` only relaxes the quorum + (never tightens), and `MIN_DISTINCT_DOMAINS` is best-effort (never fails a POST). + A drop means something else is wrong — check upstream peer health and the + `arweave_chunk_broadcast_total{status="fail"}` breakdown. +- **`chunk_post_domain_shortfall_total{below_target}` floods after raising + `MIN_DISTINCT_DOMAINS`** → your reachable peer set can't supply that many + distinct domains on fresh chunks (the propagation race). It's advisory, not a + failure; lower the target if the extra fan-out isn't worth it. +- **Egress / event-loop latency rises after raising `MIN_DISTINCT_DOMAINS`** → + expected: the breadth-seeking contacts more peers. Tune the target down. +- **A tip outage no longer shows as POST failures** (with `SOFT_FALLBACK=true`) → + intended. Confirm the fallback is carrying it via + `chunk_post_preferred_shortfall_total{reason=tips_unavailable|tips_failed}` + rising while success rate holds. + +## 8. Rollback + +Set both flags to their defaults and recreate `core`: + +``` +CHUNK_POST_MIN_DISTINCT_DOMAINS=0 +CHUNK_POST_PREFERRED_SOFT_FALLBACK=false +``` + +This reverts to the exact legacy fan-out behavior immediately on the next config +load. There is no persistent state to unwind and no data implication — the flags +only affect in-flight fan-out decisions, not stored data. + +## 9. What this change does NOT touch + +For reassurance, the following are unaffected: + +- The **optimistic ingest cache** (the gateway's local chunk storage) — unchanged. +- **Data retrieval / serving** — unchanged. +- **Default behavior** — with flags off, identical to the previous release. + +## See also + +- `docs/envs.md` — full env-var reference (the `CHUNK_POST_*` block). +- `docs/chunk-ingest-cache.md` — the separate local chunk cache. +- `docs/glossary.md` — *fault domain*, *tip node*, *chunk fan-out*. diff --git a/docs/envs.md b/docs/envs.md index 6a38cc958..c56cffa58 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -206,6 +206,8 @@ This document describes the environment variables that can be used to configure | AWS_S3_CONTIGUOUS_DATA_PREFIX | String | undefined | Prefix for the S3 bucket to organize data | | CHUNK_POST_MIN_SUCCESS_COUNT | String | "3" | Minimum count of 200 responses for of a given chunk to be considered properly seeded | | CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT | String | "2" | Minimum count of 200 responses from preferred (tip) nodes for a chunk to be considered properly seeded. Set to 0 to disable preferred node requirement | +| CHUNK_POST_MIN_DISTINCT_DOMAINS | String | "0" | Best-effort diversity target: distinct fault domains (IP /24 for v4, /48 for v6) a chunk POST should land on. 0 = disabled. Drives fan-out breadth (keeps contacting peers until the target is met or the peer set is exhausted) and emits `chunk_post_domain_shortfall_total{reason=below_target}` when unmet, but does **not** fail a POST — fresh chunks can only land on preferred nodes until the tx's data_root propagates, so gating success on it would hard-fail legitimate first-posts. Distinct-domain count is surfaced per-POST via the `X-AR-IO-Chunk-Placement-Domains` response header | +| CHUNK_POST_PREFERRED_SOFT_FALLBACK | Boolean | "false" | When true, a chunk POST meets quorum via a strong distinct-domain quorum of discovered peers **whenever the preferred (tip) quorum falls short** — whether the tips are ineligible (down/over-queue) or eligible-but-failing (e.g. a network partition of the tip /24). Removes the single-fault-domain tips SPOF. **This softens the tip requirement in steady state, not only during a full outage:** with the flag on, the effective quorum becomes `CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT` tip successes **OR** `CHUNK_POST_MIN_SUCCESS_COUNT` total successes across ≥ `max(CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT, CHUNK_POST_MIN_DISTINCT_DOMAINS)` distinct fault domains — so a normal POST where only one tip acks but several independent discovered peers land will now pass. Do **not** assume the unconditional 2-tip guarantee still holds with the flag on. `chunk_post_preferred_shortfall_total{reason}` is emitted (tips_unavailable/tips_failed) so tip shortfalls stay visible. Default false preserves the hard preferred-success requirement | | CHUNK_POST_MAX_CONSECUTIVE_FAILURES | String | "5" | Maximum consecutive 4xx responses before stopping chunk broadcast. Only applies when no peers have accepted the chunk. Set to 0 to disable early termination | | ARWEAVE_POST_DRY_RUN | Boolean | false | If true, simulates transaction header and chunk submission without posting to Arweave. `POST /tx` and `POST /chunk` return 200 OK as if successful; only the final network broadcast is skipped. Works on both port 3000 (Envoy) and port 4000 (direct). By default, transaction signatures and chunk merkle proofs are still validated before success. When disabled, Envoy routes these requests to trusted Arweave nodes instead; `GET /tx` is always proxied to the trusted node regardless of this setting. | | ARWEAVE_POST_DRY_RUN_SKIP_VALIDATION | Boolean | false | If true (and `ARWEAVE_POST_DRY_RUN` is enabled), skips transaction signature verification and chunk merkle proof validation for faster testing. | diff --git a/docs/glossary.md b/docs/glossary.md index 2aa786547..c7bc867bc 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -428,6 +428,24 @@ other gateways in the network. **Observer Wallet** - A wallet used to sign and submit observation reports about other gateways in the network, enabling decentralized monitoring. +**Chunk Fan-out (Broadcast)** - When a chunk is posted to the gateway +(`POST /chunk`), it is relayed ("fanned out") in parallel to multiple Arweave +peers so it spreads toward miners. A quorum of successful posts marks the chunk +seeded. The gateway is a seeding *relay*; durability is owned by the uploader, +which re-verifies permanence and re-seeds. See +[Chunk Fan-out Seeding](chunk-fanout-seeding.md). + +**Fault Domain** - A grouping of peers by shared failure risk, approximated by IP +subnet (`/24` for IPv4, `/48` for IPv6). Peers in the same fault domain can fail +together (same operator, rack, or network block), so counting *distinct* fault +domains — not raw peer successes — measures genuine seeding redundancy. + +**Tip Node (Preferred Chunk-Post Node)** - A preferred, mining-adjacent node the +gateway posts chunks to first (default `tip-1..5.arweave.xyz`), the on-ramp to +miners' mempools. Note the defaults all share one fault domain +(`38.29.227.0/24`), which is why fault-domain-aware fan-out treats them as a +single point of failure. + ## Additional Terms **Base64URL** - URL-safe base64 encoding used throughout Arweave for IDs, data diff --git a/monitoring/grafana/dashboards/default.json b/monitoring/grafana/dashboards/default.json index cd8b613b4..582015ce6 100644 --- a/monitoring/grafana/dashboards/default.json +++ b/monitoring/grafana/dashboards/default.json @@ -59,7 +59,9 @@ "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { - "calcs": ["lastNotNull"], + "calcs": [ + "lastNotNull" + ], "fields": "", "values": false }, @@ -1023,6 +1025,365 @@ ], "title": "I/O Utilization", "type": "timeseries" + }, + { + "id": 14, + "type": "row", + "title": "Chunk Fan-out Seeding (placement)", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 40 + }, + "panels": [] + }, + { + "id": 15, + "type": "timeseries", + "title": "Distinct fault domains per chunk POST", + "description": "Independent operators (IP /24·/48) a chunk landed on per POST. p10 is the seeding-risk tail: low p10 = a meaningful slice of chunks are landing on few fault domains.", + "datasource": "${datasource}", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 41 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "auto", + "pointSize": 5, + "stacking": { + "group": "A", + "mode": "none" + }, + "axisPlacement": "auto", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": "${datasource}", + "editorMode": "code", + "expr": "sum(rate(chunk_post_distinct_domains_sum[$__rate_interval]))/sum(rate(chunk_post_distinct_domains_count[$__rate_interval]))", + "instant": false, + "legendFormat": "avg", + "range": true, + "refId": "A" + }, + { + "datasource": "${datasource}", + "editorMode": "code", + "expr": "histogram_quantile(0.5, sum by (le) (rate(chunk_post_distinct_domains_bucket[$__rate_interval])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "B" + }, + { + "datasource": "${datasource}", + "editorMode": "code", + "expr": "histogram_quantile(0.1, sum by (le) (rate(chunk_post_distinct_domains_bucket[$__rate_interval])))", + "instant": false, + "legendFormat": "p10 (thinnest 10%)", + "range": true, + "refId": "C" + } + ] + }, + { + "id": 16, + "type": "timeseries", + "title": "Preferred (tip) shortfall rate", + "description": "POSTs where the preferred (tip) success count fell short. tips_unavailable = the tip /24 was down/over-queue (the SPOF the soft-preferred fallback covers); tips_failed = tips reachable but did not ack. Emitted even when the fallback still made the POST succeed.", + "datasource": "${datasource}", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 41 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "auto", + "pointSize": 5, + "stacking": { + "group": "A", + "mode": "none" + }, + "axisPlacement": "auto", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": "${datasource}", + "editorMode": "code", + "expr": "sum(rate(chunk_post_preferred_shortfall_total[$__rate_interval])) by (reason)", + "instant": false, + "legendFormat": "{{reason}}", + "range": true, + "refId": "A" + } + ] + }, + { + "id": 17, + "type": "timeseries", + "title": "Distinct-domain target shortfall rate", + "description": "Only when CHUNK_POST_MIN_DISTINCT_DOMAINS>0. unmet = target achievable but not reached; degraded = eligible peer set could not supply the target (soft-capped, POST not failed on scarcity).", + "datasource": "${datasource}", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 49 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "auto", + "pointSize": 5, + "stacking": { + "group": "A", + "mode": "none" + }, + "axisPlacement": "auto", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": "${datasource}", + "editorMode": "code", + "expr": "sum(rate(chunk_post_domain_shortfall_total[$__rate_interval])) by (reason)", + "instant": false, + "legendFormat": "{{reason}}", + "range": true, + "refId": "A" + } + ] + }, + { + "id": 18, + "type": "timeseries", + "title": "Chunk broadcast success rate", + "description": "Overall share of chunk broadcasts meeting the quorum verdict. Context for the diversity panels above.", + "datasource": "${datasource}", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 49 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "auto", + "pointSize": 5, + "stacking": { + "group": "A", + "mode": "none" + }, + "axisPlacement": "auto", + "axisColorMode": "text", + "scaleDistribution": { + "type": "linear" + }, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": "${datasource}", + "editorMode": "code", + "expr": "sum(rate(arweave_chunk_broadcast_total{status=\"success\"}[$__rate_interval])) / sum(rate(arweave_chunk_broadcast_total[$__rate_interval]))", + "instant": false, + "legendFormat": "success ratio", + "range": true, + "refId": "A" + } + ] } ], "refresh": "auto", diff --git a/src/arweave/composite-client.test.ts b/src/arweave/composite-client.test.ts index 91fe02824..c8548e256 100644 --- a/src/arweave/composite-client.test.ts +++ b/src/arweave/composite-client.test.ts @@ -8,7 +8,13 @@ import { strict as assert } from 'node:assert'; import { describe, it, beforeEach, afterEach, mock } from 'node:test'; import { default as Arweave } from 'arweave'; -import { ArweaveCompositeClient } from './composite-client.js'; +import { + ArweaveCompositeClient, + chunkPostPeersCacheKey, + chunkPostPeerDomain, + evaluateChunkBroadcastVerdict, + type ChunkBroadcastVerdictInput, +} from './composite-client.js'; import { UniformFailureSimulator } from '../lib/chaos.js'; import { ArweavePeerManager } from '../peers/arweave-peer-manager.js'; import log from '../log.js'; @@ -409,4 +415,228 @@ describe('ArweaveCompositeClient', () => { (client as any).peerGetChunk = originalPeerGetChunk; }); }); + + describe('chunkPostPeersCacheKey', () => { + it('distinguishes different peer sets of equal length', () => { + // The bug this fixes: a length-only key collided these onto one cached + // ordering for the full cache window, narrowing seeding diversity. + const a = ['http://a:1984', 'http://b:1984', 'http://c:1984']; + const b = ['http://x:1984', 'http://y:1984', 'http://z:1984']; + assert.equal(a.length, b.length); + assert.notEqual(chunkPostPeersCacheKey(a), chunkPostPeersCacheKey(b)); + }); + + it('is order-independent for the same set (set identity)', () => { + const ordered = ['http://a:1984', 'http://b:1984', 'http://c:1984']; + const shuffled = ['http://c:1984', 'http://a:1984', 'http://b:1984']; + assert.equal( + chunkPostPeersCacheKey(ordered), + chunkPostPeersCacheKey(shuffled), + ); + }); + + it('does not mutate the input array', () => { + const peers = ['http://c:1984', 'http://a:1984', 'http://b:1984']; + const snapshot = [...peers]; + chunkPostPeersCacheKey(peers); + assert.deepEqual(peers, snapshot); + }); + + it('handles the empty set', () => { + assert.equal(chunkPostPeersCacheKey([]), ''); + }); + }); +}); + +describe('chunkPostPeerDomain', () => { + it('buckets IP-literal peers by /24', () => { + assert.equal( + chunkPostPeerDomain('http://38.29.227.74:1984'), + '38.29.227.0/24', + ); + }); + + it('collapses the resolved tip nodes into one domain', () => { + const tips = [ + 'http://38.29.227.74:1984', + 'http://38.29.227.75:1984', + 'http://38.29.227.70:1984', + ]; + assert.equal(new Set(tips.map(chunkPostPeerDomain)).size, 1); + }); + + it('handles bracketed IPv6 hosts', () => { + assert.equal( + chunkPostPeerDomain('http://[2001:db8:abcd:1234::1]:1984'), + '2001:0db8:abcd::/48', + ); + }); + + it('falls back to the peer string for unparseable input', () => { + assert.equal(chunkPostPeerDomain('not a url'), 'not a url'); + }); +}); + +describe('evaluateChunkBroadcastVerdict', () => { + const base: ChunkBroadcastVerdictInput = { + successCount: 0, + preferredSuccessCount: 0, + distinctDomainCount: 0, + preferredEligibleCount: 5, + minSuccessCount: 3, + minPreferredSuccessCount: 2, + minDistinctDomains: 0, + preferredSoftFallback: false, + }; + + describe('defaults reduce to legacy behavior', () => { + it('succeeds exactly when success>=min AND preferred>=minPreferred', () => { + // Meets both -> succeeded + assert.equal( + evaluateChunkBroadcastVerdict({ + ...base, + successCount: 3, + preferredSuccessCount: 2, + }).succeeded, + true, + ); + // Preferred short -> fails (legacy hard requirement preserved) + assert.equal( + evaluateChunkBroadcastVerdict({ + ...base, + successCount: 5, + preferredSuccessCount: 1, + }).succeeded, + false, + ); + // Success short -> fails + assert.equal( + evaluateChunkBroadcastVerdict({ + ...base, + successCount: 2, + preferredSuccessCount: 2, + }).succeeded, + false, + ); + }); + + it('does not apply the domain target when feature is off', () => { + assert.equal( + evaluateChunkBroadcastVerdict({ + ...base, + successCount: 3, + preferredSuccessCount: 2, + distinctDomainCount: 1, // all one /24, but feature off -> fine + }).succeeded, + true, + ); + }); + }); + + describe('soft-preferred fallback', () => { + it('rescues a tips-down POST via a strong distinct-domain quorum', () => { + const v = evaluateChunkBroadcastVerdict({ + ...base, + preferredSoftFallback: true, + preferredEligibleCount: 0, // tips unavailable (ineligible) + successCount: 4, + preferredSuccessCount: 0, + distinctDomainCount: 3, // >= max(minPreferred=2, minDomains=0) + }); + assert.equal(v.succeeded, true); + assert.equal(v.preferredShortfall, 'tips_unavailable'); + }); + + it('ALSO fires when tips are eligible-but-failing (the soak trace)', () => { + // Regression for the #812 soak finding: tip /24 blocked, one surviving + // preferred node in another domain (preferredEligibleCount>0, + // preferredSuccessCount=1<2), 14 independent domains seeded -> must SUCCEED. + const v = evaluateChunkBroadcastVerdict({ + ...base, + preferredSoftFallback: true, + preferredEligibleCount: 5, // tips eligible, just partitioned/failing + successCount: 19, + preferredSuccessCount: 1, + distinctDomainCount: 14, + }); + assert.equal(v.succeeded, true); + assert.equal(v.preferredShortfall, 'tips_failed'); + }); + + it('requires a sufficiently diverse fallback quorum', () => { + const v = evaluateChunkBroadcastVerdict({ + ...base, + preferredSoftFallback: true, + preferredEligibleCount: 5, + successCount: 4, + preferredSuccessCount: 0, + distinctDomainCount: 1, // one /24 -> not a real stand-in for the tips + }); + assert.equal(v.succeeded, false); + }); + + it('does not fire when the preferred quorum is actually met', () => { + // Healthy tips: normal path succeeds, fallback is irrelevant. + const v = evaluateChunkBroadcastVerdict({ + ...base, + preferredSoftFallback: true, + successCount: 3, + preferredSuccessCount: 2, + distinctDomainCount: 1, + }); + assert.equal(v.succeeded, true); + assert.equal(v.preferredShortfall, 'none'); + }); + + it('stays off unless the flag is set', () => { + const v = evaluateChunkBroadcastVerdict({ + ...base, + preferredSoftFallback: false, + preferredEligibleCount: 0, + successCount: 4, + distinctDomainCount: 4, + }); + assert.equal(v.succeeded, false); + }); + }); + + describe('distinct-domain target is best-effort (never hard-fails)', () => { + it('does NOT fail a POST that meets base quorum but misses the domain target', () => { + // Fresh-chunk propagation race: only preferred domains accept. Must still + // succeed (gating on the target would hard-fail legitimate first-posts) — + // but the shortfall is surfaced as a metric. + const v = evaluateChunkBroadcastVerdict({ + ...base, + minDistinctDomains: 3, + successCount: 3, + preferredSuccessCount: 2, + distinctDomainCount: 2, // below target + }); + assert.equal(v.succeeded, true); + assert.equal(v.domainShortfall, 'below_target'); + }); + + it('reports no shortfall when the target is met', () => { + const v = evaluateChunkBroadcastVerdict({ + ...base, + minDistinctDomains: 3, + successCount: 4, + preferredSuccessCount: 2, + distinctDomainCount: 3, + }); + assert.equal(v.succeeded, true); + assert.equal(v.domainShortfall, 'none'); + }); + + it('still fails when the BASE quorum is not met, regardless of domains', () => { + const v = evaluateChunkBroadcastVerdict({ + ...base, + minDistinctDomains: 3, + successCount: 2, // below min success + preferredSuccessCount: 2, + distinctDomainCount: 5, + }); + assert.equal(v.succeeded, false); + }); + }); }); diff --git a/src/arweave/composite-client.ts b/src/arweave/composite-client.ts index 9ca2295e9..4295eb3de 100644 --- a/src/arweave/composite-client.ts +++ b/src/arweave/composite-client.ts @@ -21,6 +21,7 @@ import { LRUCache } from 'lru-cache'; import { FailureSimulator } from '../lib/chaos.js'; import { fromB64Url } from '../lib/encoding.js'; +import { ipFaultDomain } from '../lib/ip-utils.js'; import { sanityCheckBlock, sanityCheckChunk, @@ -156,6 +157,139 @@ interface PeerChunkQueue { totalSuccesses: number; } +/** + * Stable memo key for the chunk-POST peer-sort cache. Identifies the *set* of + * eligible peers independent of order, so two different peer sets of equal + * length no longer collide on a shared cached ordering (the prior length-only + * key did, narrowing seeding diversity within the cache window). + * + * Cheap by design: one sort + join over the bounded postChunk peer set, far + * less work than the weighted selection the memo guards. Does not mutate the + * caller's array (`slice` before `sort`). + */ +export function chunkPostPeersCacheKey(peers: readonly string[]): string { + return peers.slice().sort().join(','); +} + +/** + * Reduce a chunk-POST peer URL to its fault-domain bucket (IP /24 v4, /48 v6). + * The postChunk peer list is IP-literal after DNS resolution, so this needs no + * DNS. An unparseable URL (or a host that can't be bucketed) falls back to the + * peer string itself, counting it as its own domain rather than collapsing. + */ +export function chunkPostPeerDomain(peer: string): string { + try { + // URL.hostname wraps IPv6 in brackets (e.g. "[::1]"); strip them. + const host = new URL(peer).hostname.replace(/^\[|\]$/g, ''); + return ipFaultDomain(host); + } catch { + return peer; + } +} + +export interface ChunkBroadcastVerdictInput { + successCount: number; + preferredSuccessCount: number; + /** Distinct fault domains among successful posts. */ + distinctDomainCount: number; + /** + * How many preferred (tip) peers were eligible. Used only to LABEL the + * shortfall reason (tips_unavailable vs tips_failed) — it no longer gates the + * fallback (see evaluateChunkBroadcastVerdict). + */ + preferredEligibleCount: number; + minSuccessCount: number; + minPreferredSuccessCount: number; + /** CHUNK_POST_MIN_DISTINCT_DOMAINS (0 = feature off). */ + minDistinctDomains: number; + /** CHUNK_POST_PREFERRED_SOFT_FALLBACK. */ + preferredSoftFallback: boolean; +} + +export interface ChunkBroadcastVerdict { + succeeded: boolean; + preferredShortfall: 'none' | 'tips_unavailable' | 'tips_failed'; + domainShortfall: 'none' | 'below_target'; +} + +/** + * Centralized chunk-broadcast quorum verdict — the single source of truth for + * "did this POST seed sufficiently?", consumed by both the broadcaster and (via + * the returned `succeeded`) the POST /chunk handler. + * + * With both feature flags at their defaults (`minDistinctDomains = 0`, + * `preferredSoftFallback = false`) this reduces EXACTLY to the legacy rule + * `successCount ≥ min AND preferredSuccessCount ≥ minPreferred` (see the + * regression test). The new behavior is additive and opt-in: + * - **distinct-domain target**: also require N distinct domains among successes, + * but soft — capped by what the eligible set can supply, so it never hard-fails + * on domain scarcity. + * - **soft-preferred fallback**: when NO tip was eligible (the tips-down SPOF), + * accept a strong distinct-domain discovered quorum in lieu of the preferred + * requirement. + * + * Shortfall labels are advisory (for metrics) and never fail a POST on their own. + */ +export function evaluateChunkBroadcastVerdict( + i: ChunkBroadcastVerdictInput, +): ChunkBroadcastVerdict { + const meetsSuccess = i.successCount >= i.minSuccessCount; + const meetsPreferred = i.preferredSuccessCount >= i.minPreferredSuccessCount; + + // Soft-preferred fallback: fire whenever the preferred quorum falls short + // (tips down OR eligible-but-failing) and a strong distinct-domain discovered + // quorum stands in for them. + // + // Note this softens the preferred requirement in STEADY STATE too, not only + // during a full outage: with the flag on, the effective quorum is + // preferred >= minPreferred OR (success >= minSuccess AND domains >= fallbackTarget). + // A normal POST where only one tip acks but several independent discovered + // peers land will pass — the intended availability win; the unconditional + // "minPreferred tips" guarantee no longer holds when the flag is on. + // + // NOTE: previously gated on `preferredEligibleCount === 0`, which only holds + // when the tip per-peer queues saturate past the depth threshold (or no tips + // are configured). A real tip partition leaves them eligible-but-failing, so + // the fallback never fired for the exact outage it was built for — and any one + // surviving preferred node in another domain kept eligibleCount ≥ 1 forever. + // The live soak on #812 caught this. `!meetsPreferred` is the right trigger: + // it can only be true after we genuinely could not reach the preferred quorum + // (tips are dispatched first, so a healthy tip set meets it via the normal + // path before this is consulted). + const fallbackTarget = Math.max( + i.minPreferredSuccessCount, + i.minDistinctDomains, + ); + const meetsFallback = + i.preferredSoftFallback && + meetsSuccess && + !meetsPreferred && + i.distinctDomainCount >= fallbackTarget; + + // The distinct-domain target is a best-effort diversity GOAL, not a hard gate + // on success. Fresh chunks can only land on the preferred (upload-oriented) + // nodes until the tx's data_root propagates — discovered peers reject an + // unknown data_root — so gating success on N distinct domains would hard-fail + // legitimate first-posts during that race. Instead the target drives fan-out + // breadth (early termination) and a shortfall metric; broader diversity + // accrues over the seeding lifecycle (re-posts as the tx propagates), not on a + // single POST. + const succeeded = meetsSuccess && (meetsPreferred || meetsFallback); + + let preferredShortfall: ChunkBroadcastVerdict['preferredShortfall'] = 'none'; + if (!meetsPreferred) { + preferredShortfall = + i.preferredEligibleCount === 0 ? 'tips_unavailable' : 'tips_failed'; + } + + const domainShortfall: ChunkBroadcastVerdict['domainShortfall'] = + i.minDistinctDomains > 0 && i.distinctDomainCount < i.minDistinctDomains + ? 'below_target' + : 'none'; + + return { succeeded, preferredShortfall, domainShortfall }; +} + export class ArweaveCompositeClient implements ChainSource, @@ -311,11 +445,13 @@ export class ArweaveCompositeClient }, { maxAge: config.CHUNK_POST_SORTED_PEERS_CACHE_DURATION_MS, - // Use array length as cache key for O(1) performance. This means different - // peer lists of the same length will share cached results, which is acceptable - // because: 1) peer weights change gradually, 2) the cache duration is short (10s), - // and 3) this avoids expensive operations on every chunk POST request. - normalizer: (args) => args[0].length.toString(), + // Key by the *set* of eligible peers (order-independent), not its length. + // The prior length-only key collided distinct peer sets of equal size onto + // one cached ordering for the full 10s window, quietly narrowing seeding + // diversity. This key is still cheap — a sort+join over the bounded + // postChunk peer set, far less than the weighted selection it guards — and + // the 10s maxAge still absorbs gradual weight drift. + normalizer: (args) => chunkPostPeersCacheKey(args[0]), }, ); @@ -1872,6 +2008,9 @@ export class ArweaveCompositeClient successCount: 0, preferredSuccessCount: 0, failureCount: 0, + distinctDomainCount: 0, + preferredEligibleCount: 0, + succeeded: false, results: [], }; } @@ -1891,6 +2030,15 @@ export class ArweaveCompositeClient ); const shuffledPeers = [...shuffleArray(preferred), ...nonPreferred]; + // Fault-domain accounting (B). `preferredEligibleCount` labels the + // shortfall reason; `successDomains` tracks the distinct fault domains a + // chunk actually landed on; `preferredAttempted` lets the fallback early- + // terminate only after every tip has had its shot (so a healthy tip set is + // never skipped, but a tips outage doesn't force a walk of the whole peer + // list — the latency half of the soak feedback). + const preferredEligibleCount = preferred.length; + const successDomains = new Set(); + // 3. Broadcast in parallel with concurrency limit const peerConcurrencyLimit = pLimit(config.CHUNK_POST_PEER_CONCURRENCY); @@ -1901,11 +2049,19 @@ export class ArweaveCompositeClient // counts may also be slightly inaccurate; use the results array for precise values. let successCount = 0; let preferredSuccessCount = 0; + let preferredAttempted = 0; let failureCount = 0; let consecutive4xxFailures = 0; let hasAnySuccess = false; const results: BroadcastChunkResponses[] = []; + // Fallback quorum used for early termination (mirrors the verdict's + // fallbackTarget). Only relevant when the soft-preferred fallback is on. + const earlyFallbackTarget = Math.max( + chunkPostMinPreferredSuccessCount, + config.CHUNK_POST_MIN_DISTINCT_DOMAINS, + ); + this.log.debug('Starting chunk broadcast', { eligiblePeers: eligiblePeers.length, sortedPeers: sortedPeers.length, @@ -1919,11 +2075,37 @@ export class ArweaveCompositeClient // Create promises for all peers const peerPromises = shuffledPeers.map((peer) => peerConcurrencyLimit(async () => { - // Skip if we already have enough successes (both overall and preferred) - if ( + // Terminate early if the broadcast is already satisfied, by EITHER: + // (a) the normal quorum — overall + preferred + (opt-in) distinct + // domains. The distinct-domain term is 0 by default, so this + // reduces to the legacy condition unless the operator opts in; OR + // (b) the soft-preferred fallback — every tip has been attempted and + // the preferred quorum still fell short, but we already have the + // overall + distinct-domain fallback quorum. Without this, a tips + // outage can never early-terminate (preferred quorum is + // unreachable) and every POST walks the whole peer list (~12s in + // the soak). Gated on preferredAttempted so a healthy tip set is + // never skipped. + // + // This intentionally does NOT reuse evaluateChunkBroadcastVerdict: + // both branches here are STRICTER than the verdict's success rule + // (normal keeps the distinct-domain term the verdict dropped, to drive + // diversity-seeking breadth when MIN_DISTINCT_DOMAINS > 0; fallback + // adds the preferredAttempted gate). Each branch provably implies + // verdict.succeeded, so we never terminate before the POST would + // succeed — we only (deliberately) keep seeding for breadth past it. + // Mirroring the verdict here would remove the breadth-seeking feature. + const normalQuorumMet = successCount >= chunkPostMinSuccessCount && - preferredSuccessCount >= chunkPostMinPreferredSuccessCount - ) { + preferredSuccessCount >= chunkPostMinPreferredSuccessCount && + successDomains.size >= config.CHUNK_POST_MIN_DISTINCT_DOMAINS; + const fallbackQuorumMet = + config.CHUNK_POST_PREFERRED_SOFT_FALLBACK && + preferredAttempted >= preferred.length && + preferredSuccessCount < chunkPostMinPreferredSuccessCount && + successCount >= chunkPostMinSuccessCount && + successDomains.size >= earlyFallbackTarget; + if (normalQuorumMet || fallbackQuorumMet) { this.log.debug('Skipping peer due to success threshold reached', { peer, }); @@ -1962,6 +2144,14 @@ export class ArweaveCompositeClient }; } + // Count this as a preferred attempt (success or failure) so the + // fallback early-termination only engages once every tip has had its + // shot. Counted here — past the skip guards — so skipped peers don't. + const isPreferred = this.peerManager.isPreferredChunkPostPeer(peer); + if (isPreferred) { + preferredAttempted++; + } + try { const result = await this.queueChunkPost( peer, @@ -1976,6 +2166,7 @@ export class ArweaveCompositeClient if (result.success) { successCount++; + successDomains.add(chunkPostPeerDomain(peer)); if (this.peerManager.isPreferredChunkPostPeer(peer)) { preferredSuccessCount++; } @@ -2050,10 +2241,43 @@ export class ArweaveCompositeClient span.setAttribute('chunk.broadcast.failure_count', failureCount); span.setAttribute('chunk.broadcast.total_results', results.length); - const succeeded = - successCount >= chunkPostMinSuccessCount && - preferredSuccessCount >= chunkPostMinPreferredSuccessCount; + const distinctDomainCount = successDomains.size; + const verdict = evaluateChunkBroadcastVerdict({ + successCount, + preferredSuccessCount, + distinctDomainCount, + preferredEligibleCount, + minSuccessCount: chunkPostMinSuccessCount, + minPreferredSuccessCount: chunkPostMinPreferredSuccessCount, + minDistinctDomains: config.CHUNK_POST_MIN_DISTINCT_DOMAINS, + preferredSoftFallback: config.CHUNK_POST_PREFERRED_SOFT_FALLBACK, + }); + const succeeded = verdict.succeeded; span.setAttribute('chunk.broadcast.succeeded', succeeded); + span.setAttribute( + 'chunk.broadcast.distinct_domains', + distinctDomainCount, + ); + span.setAttribute( + 'chunk.broadcast.preferred_eligible', + preferredEligibleCount, + ); + + // Placement-evidence observability (B). The histogram answers "across how + // many independent fault domains did this chunk land?"; the shortfall + // counters stay visible even when the soft fallback makes the POST succeed, + // so a tips outage is never silently masked. + metrics.chunkPostDistinctDomainsHistogram.observe(distinctDomainCount); + if (verdict.preferredShortfall !== 'none') { + metrics.chunkPostPreferredShortfallCounter.inc({ + reason: verdict.preferredShortfall, + }); + } + if (verdict.domainShortfall !== 'none') { + metrics.chunkPostDomainShortfallCounter.inc({ + reason: verdict.domainShortfall, + }); + } // Determine early termination reason // Only report 'consecutive_failures' when: @@ -2104,6 +2328,9 @@ export class ArweaveCompositeClient successCount, preferredSuccessCount, failureCount, + distinctDomainCount, + preferredEligibleCount, + succeeded, results, }; } catch (error: any) { diff --git a/src/config.ts b/src/config.ts index 1e4e0e40f..16220efa5 100644 --- a/src/config.ts +++ b/src/config.ts @@ -804,6 +804,26 @@ export const CHUNK_POST_PEER_CONCURRENCY = +env.varOrDefault( String(CHUNK_POST_MIN_SUCCESS_COUNT), ); +// Best-effort distinct-fault-domain target (IP /24 v4, /48 v6) for a chunk POST. +// 0 = disabled (fault-domain-blind, the legacy behavior). It drives fan-out +// breadth + a shortfall metric but never fails a POST — see +// evaluateChunkBroadcastVerdict. Parsed as a non-negative integer so a bad env +// value (NaN/negative/fraction) can't leak into the verdict and skew the +// threshold. +export const CHUNK_POST_MIN_DISTINCT_DOMAINS = env.nonNegativeIntOrDefault( + 'CHUNK_POST_MIN_DISTINCT_DOMAINS', + 0, +); + +// When true, a chunk POST may meet quorum via a strong distinct-domain quorum of +// discovered peers when NO preferred (tip) peer was even eligible — i.e. the tips +// are down/over-queue. Removes the single-fault-domain tips SPOF. Default false +// preserves the hard preferred-success requirement. +export const CHUNK_POST_PREFERRED_SOFT_FALLBACK = + env + .varOrDefault('CHUNK_POST_PREFERRED_SOFT_FALLBACK', 'false') + .toLowerCase() === 'true'; + // Maximum consecutive 4xx failures before stopping broadcast (0 to disable) export const CHUNK_POST_MAX_CONSECUTIVE_FAILURES = +env.varOrDefault( 'CHUNK_POST_MAX_CONSECUTIVE_FAILURES', diff --git a/src/constants.ts b/src/constants.ts index 4280003d8..9ab37c1ea 100644 --- a/src/constants.ts +++ b/src/constants.ts @@ -46,6 +46,9 @@ export const headerNames = { chunkTxPath: 'X-Arweave-Chunk-Tx-Path', chunkTxId: 'X-Arweave-Chunk-Tx-Id', chunkTxStartOffset: 'X-Arweave-Chunk-Tx-Start-Offset', + // Placement-evidence contract: distinct fault domains (IP /24·/48) a POSTed + // chunk landed on, surfaced on the POST /chunk response for the poster. + chunkPlacementDomains: 'X-AR-IO-Chunk-Placement-Domains', rootTransactionId: 'X-AR-IO-Root-Transaction-Id', rootPath: 'X-AR-IO-Root-Path', rootItemOffset: 'X-AR-IO-Root-Item-Offset', diff --git a/src/data/rebroadcasting-chunk-source.test.ts b/src/data/rebroadcasting-chunk-source.test.ts index 895654035..aabd155e8 100644 --- a/src/data/rebroadcasting-chunk-source.test.ts +++ b/src/data/rebroadcasting-chunk-source.test.ts @@ -95,6 +95,9 @@ class MockChunkBroadcaster implements ChunkBroadcaster { successCount: 1, preferredSuccessCount: 0, failureCount: 0, + distinctDomainCount: 1, + preferredEligibleCount: 0, + succeeded: true, results: [ { peer: 'http://mock-peer', @@ -151,7 +154,11 @@ class MockChunkBroadcaster implements ChunkBroadcaster { this.broadcastPromise = null; this.result = { successCount: 1, + preferredSuccessCount: 0, failureCount: 0, + distinctDomainCount: 1, + preferredEligibleCount: 0, + succeeded: true, results: [ { peer: 'http://mock-peer', @@ -379,7 +386,11 @@ describe('RebroadcastingChunkSource', () => { it('should not cache when success count below threshold', async () => { mockBroadcaster.result = { successCount: 0, + preferredSuccessCount: 0, failureCount: 1, + distinctDomainCount: 0, + preferredEligibleCount: 0, + succeeded: false, results: [ { peer: 'http://mock-peer', @@ -406,7 +417,11 @@ describe('RebroadcastingChunkSource', () => { // Fix broadcaster result mockBroadcaster.result = { successCount: 1, + preferredSuccessCount: 0, failureCount: 0, + distinctDomainCount: 1, + preferredEligibleCount: 0, + succeeded: true, results: [ { peer: 'http://mock-peer', @@ -473,6 +488,9 @@ describe('RebroadcastingChunkSource', () => { successCount: 1, preferredSuccessCount: 0, failureCount: 0, + distinctDomainCount: 1, + preferredEligibleCount: 0, + succeeded: true, results: [ { peer: 'http://mock-peer', diff --git a/src/lib/ip-utils.test.ts b/src/lib/ip-utils.test.ts index c7e68b8df..19dc76aa9 100644 --- a/src/lib/ip-utils.test.ts +++ b/src/lib/ip-utils.test.ts @@ -15,6 +15,8 @@ import { isIpInCidr, isAnyIpAllowlisted, isAnyIpBlocked, + expandIpv6, + ipFaultDomain, } from './ip-utils.js'; describe('IP Utilities', () => { @@ -389,4 +391,85 @@ describe('IP Utilities', () => { ); }); }); + + describe('expandIpv6', () => { + it('expands full and compressed forms to 8 padded groups', () => { + assert.deepEqual(expandIpv6('2001:db8::1'), [ + '2001', + '0db8', + '0000', + '0000', + '0000', + '0000', + '0000', + '0001', + ]); + assert.deepEqual(expandIpv6('::'), Array(8).fill('0000')); + assert.deepEqual(expandIpv6('::1'), [...Array(7).fill('0000'), '0001']); + }); + + it('returns undefined for non-IPv6 / malformed input', () => { + assert.equal(expandIpv6('1.2.3.4'), undefined); + assert.equal(expandIpv6('2001::db8::1'), undefined); // two `::` + assert.equal(expandIpv6('2001:db8:zz::1'), undefined); // bad hex + }); + }); + + describe('ipFaultDomain', () => { + it('collapses all five tip nodes into one /24', () => { + const tips = [ + '38.29.227.74', + '38.29.227.75', + '38.29.227.76', + '38.29.227.69', + '38.29.227.70', + ]; + const domains = new Set(tips.map((ip) => ipFaultDomain(ip))); + assert.equal(domains.size, 1); + assert.equal([...domains][0], '38.29.227.0/24'); + }); + + it('keeps distinct /24s distinct', () => { + assert.notEqual( + ipFaultDomain('38.29.227.74'), + ipFaultDomain('38.29.228.74'), + ); + }); + + it('normalizes IPv4-mapped IPv6 to the IPv4 /24', () => { + assert.equal(ipFaultDomain('::ffff:38.29.227.74'), '38.29.227.0/24'); + }); + + it('buckets IPv6 to /48 by default', () => { + assert.equal( + ipFaultDomain('2001:db8:abcd:1234::1'), + '2001:0db8:abcd::/48', + ); + // same /48, different lower bits -> same bucket + assert.equal( + ipFaultDomain('2001:db8:abcd:9999::abcd'), + ipFaultDomain('2001:db8:abcd:1234::1'), + ); + // different /48 -> different bucket + assert.notEqual( + ipFaultDomain('2001:db8:abce::1'), + ipFaultDomain('2001:db8:abcd::1'), + ); + }); + + it('honors custom prefix widths', () => { + assert.equal( + ipFaultDomain('38.29.227.74', { v4Bits: 16 }), + '38.29.0.0/16', + ); + }); + + it('treats an unresolved hostname as its own domain', () => { + assert.equal(ipFaultDomain('tip-1.arweave.xyz'), 'tip-1.arweave.xyz'); + assert.notEqual( + ipFaultDomain('tip-1.arweave.xyz'), + ipFaultDomain('tip-2.arweave.xyz'), + ); + }); + }); }); diff --git a/src/lib/ip-utils.ts b/src/lib/ip-utils.ts index 48ec3268f..4725fa5da 100644 --- a/src/lib/ip-utils.ts +++ b/src/lib/ip-utils.ts @@ -229,6 +229,102 @@ export function isIpInCidr(ip: string, cidr: string): boolean { } } +/** + * Expand an IPv6 address to its 8 zero-padded 16-bit hex groups, or undefined if + * it cannot be parsed. Handles `::` zero-compression. ip-utils otherwise only + * does basic IPv6 validation; this is the minimum needed for prefix bucketing + * (see {@link ipFaultDomain}). + */ +export function expandIpv6(ip: string): string[] | undefined { + if (!ip.includes(':')) return undefined; + const halves = ip.split('::'); + if (halves.length > 2) return undefined; // at most one `::` + + const head = halves[0] ? halves[0].split(':') : []; + const tail = halves.length === 2 && halves[1] ? halves[1].split(':') : []; + + let groups: string[]; + if (halves.length === 2) { + const missing = 8 - (head.length + tail.length); + if (missing < 0) return undefined; + groups = [...head, ...Array(missing).fill('0'), ...tail]; + } else { + groups = head; + } + if (groups.length !== 8) return undefined; + + const out: string[] = []; + for (const g of groups) { + if (!/^[0-9a-fA-F]{1,4}$/.test(g)) return undefined; + out.push(parseInt(g, 16).toString(16).padStart(4, '0')); + } + return out; +} + +/** + * Reduce a peer host to a stable "fault domain" bucket key — the network block + * an address belongs to — for seeding-diversity accounting. Two peers in the + * same bucket count as one fault domain (e.g. the five `tip-*.arweave.xyz` nodes + * all resolve into `38.29.227.0/24`). + * + * - IPv4 (incl. IPv4-mapped IPv6): masked to /`v4Bits` (default 24) → `"a.b.c.0/24"`. + * - IPv6: expanded + truncated to the first `v6Bits` (default 48) → `"2001:db8:abcd::/48"`. + * - Not a valid IP (e.g. an unresolved hostname): returned verbatim, so it counts + * as its own domain rather than silently collapsing distinct hosts. + * + * Pure and hot-path cheap; performs no DNS. Callers pass the already-resolved + * peer host (the chunk-POST peer list is IP-literal after DNS resolution). + */ +export function ipFaultDomain( + host: string, + { v4Bits = 24, v6Bits = 48 }: { v4Bits?: number; v6Bits?: number } = {}, +): string { + const normalized = normalizeIpv4MappedIpv6(host.trim()); + + // IPv4 + const ipv4Segment = '(?:25[0-5]|2[0-4]\\d|1?\\d?\\d)'; + const ipv4Regex = new RegExp( + `^${ipv4Segment}\\.${ipv4Segment}\\.${ipv4Segment}\\.${ipv4Segment}$`, + ); + if (ipv4Regex.test(normalized)) { + const bits = Math.max(0, Math.min(32, v4Bits)); + const ipInt = + normalized + .split('.') + .reduce((acc, oct) => (acc << 8) + parseInt(oct, 10), 0) >>> 0; + const mask = bits === 0 ? 0 : (0xffffffff << (32 - bits)) >>> 0; + const net = (ipInt & mask) >>> 0; + const octets = [ + (net >>> 24) & 0xff, + (net >>> 16) & 0xff, + (net >>> 8) & 0xff, + net & 0xff, + ].join('.'); + return `${octets}/${bits}`; + } + + // IPv6 + if (normalized.includes(':')) { + const groups = expandIpv6(normalized); + if (groups !== undefined) { + const bits = Math.max(0, Math.min(128, v6Bits)); + const keptGroups = Math.ceil(bits / 16); + const masked = groups.slice(0, keptGroups).map((g, i) => { + const groupHigh = (i + 1) * 16; + if (groupHigh <= bits) return g; // group fully inside the prefix + const groupBits = bits - i * 16; // partial group: 1..15 bits kept + const m = groupBits === 0 ? 0 : (0xffff << (16 - groupBits)) & 0xffff; + return ((parseInt(g, 16) & m) >>> 0).toString(16).padStart(4, '0'); + }); + const prefix = masked.join(':'); + return keptGroups < 8 ? `${prefix}::/${bits}` : `${prefix}/${bits}`; + } + } + + // Not a parseable IP (unresolved hostname, garbage): its own domain. + return host.trim(); +} + /** * Check if any IP in a list matches any entry in an allowlist (supports CIDR) * @param clientIps - Array of client IP addresses to check diff --git a/src/metrics.ts b/src/metrics.ts index 9cc38df0f..17179f372 100644 --- a/src/metrics.ts +++ b/src/metrics.ts @@ -556,6 +556,35 @@ export const arweaveChunkBroadcastCounter = new promClient.Counter({ labelNames: ['status'], }); +// Distinct fault domains (IP /24 v4, /48 v6) a chunk landed on per broadcast — +// the placement-diversity signal. Buckets chosen to make "all successes in one +// network block" (1) clearly distinguishable from healthy spread. +export const chunkPostDistinctDomainsHistogram = new promClient.Histogram({ + name: 'chunk_post_distinct_domains', + help: 'Distinct fault domains (IP /24·/48) among successful chunk POSTs per broadcast', + buckets: [0, 1, 2, 3, 4, 5, 6, 8, 10], +}); + +// Preferred (tip) quorum fell short. Emitted even when the soft fallback makes +// the POST succeed, so an arweave.xyz tips outage stays visible. +// reason=tips_unavailable -> no tip peer was eligible (down/over-queue) +// reason=tips_failed -> tips were eligible but did not reach the threshold +export const chunkPostPreferredShortfallCounter = new promClient.Counter({ + name: 'chunk_post_preferred_shortfall_total', + help: 'Chunk POSTs whose preferred (tip) success count fell below the threshold', + labelNames: ['reason'], +}); + +// Distinct-domain target not met (only when CHUNK_POST_MIN_DISTINCT_DOMAINS > 0). +// The target is best-effort — it never fails a POST (see +// evaluateChunkBroadcastVerdict) — so this is a pure diversity signal. +// reason=below_target -> landed on fewer distinct domains than the target +export const chunkPostDomainShortfallCounter = new promClient.Counter({ + name: 'chunk_post_domain_shortfall_total', + help: 'Chunk POSTs that landed on fewer distinct fault domains than the target (advisory)', + labelNames: ['reason'], +}); + export const arweavePeerChunkQueuesGauge = new promClient.Gauge({ name: 'arweave_peer_chunk_queues_size', help: 'Number of peer chunk queues in memory', diff --git a/src/routes/chunk/handlers.ts b/src/routes/chunk/handlers.ts index c01e6afac..dece3dda1 100644 --- a/src/routes/chunk/handlers.ts +++ b/src/routes/chunk/handlers.ts @@ -870,19 +870,26 @@ export const createChunkPostHandler = ({ parentSpan: span, }); - // Set common broadcast span attributes - const meetsSuccessThreshold = - result.successCount >= CHUNK_POST_MIN_SUCCESS_COUNT && - result.preferredSuccessCount >= - CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT; + // The broadcaster owns the quorum verdict (success/preferred/distinct- + // domain + soft-preferred fallback) — read it, don't recompute it here. + const meetsSuccessThreshold = result.succeeded; span.setAttributes({ 'chunk.broadcast.success': meetsSuccessThreshold, 'chunk.broadcast.success_count': result.successCount, 'chunk.broadcast.preferred_success_count': result.preferredSuccessCount, + 'chunk.broadcast.distinct_domains': result.distinctDomainCount, 'chunk.broadcast.failure_count': result.failureCount, }); + // Placement-evidence contract: surface the distinct-fault-domain count + // so a caller (the bundler) can read where the chunk landed without + // parsing the body. The full result is also returned in the body below. + res.setHeader( + headerNames.chunkPlacementDomains, + String(result.distinctDomainCount), + ); + if (meetsSuccessThreshold) { span.setAttribute('http.status_code', 200); res.status(200).send(result); diff --git a/src/types.d.ts b/src/types.d.ts index e248cd6fd..2f464d3a9 100644 --- a/src/types.d.ts +++ b/src/types.d.ts @@ -801,6 +801,15 @@ interface BroadcastChunkResult { successCount: number; preferredSuccessCount: number; failureCount: number; + // Number of distinct fault domains (IP /24 v4, /48 v6) among successful posts. + // Part of the placement-evidence contract surfaced on the POST /chunk response. + distinctDomainCount: number; + // Number of preferred (tip) peers that were eligible for this broadcast. 0 means + // the tips were down/over-queue — the condition the soft-preferred fallback uses. + preferredEligibleCount: number; + // Centralized quorum verdict (success/preferred/distinct-domain + soft fallback). + // The handler reads this instead of recomputing the thresholds. + succeeded: boolean; results: BroadcastChunkResponses[]; }