Skip to content

feat: chunk fan-out fault-domain seeding (F + B1 + B2 + dashboard) #812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vilenarios wants to merge 11 commits into
base: develop
Choose a base branch
from
Open

feat: chunk fan-out fault-domain seeding (F + B1 + B2 + dashboard) #812

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d43b86b
fix(chunk-post): key sorted-peer memo by set, not length
vilenarios Jun 29, 2026
c07d6db
feat(ip-utils): add ipFaultDomain for peer network-block bucketing
vilenarios Jun 29, 2026
ce714b7
feat(chunk-post): fault-domain-aware quorum + soft-preferred fallback
vilenarios Jun 29, 2026
04eeeb0
feat(monitoring): add chunk fan-out placement panels to default dashb…
vilenarios Jul 1, 2026
38030b3
merge: B1 ipFaultDomain + B2 fault-domain quorum
vilenarios Jul 1, 2026
6251bb3
merge: F sorted-peer memo set-key
vilenarios Jul 1, 2026
8a3cd2e
merge: Grafana chunk fan-out placement panels
vilenarios Jul 1, 2026
20b77be
fix(chunk-post): address soak findings — fallback trigger + domain-ta…
vilenarios Jul 1, 2026
4e1b406
docs(chunk-post): clarify SOFT_FALLBACK softens the tip requirement i…
vilenarios Jul 1, 2026
3ba40e7
fix(chunk-post): address CodeRabbit review on #812
vilenarios Jul 1, 2026
88a6519
docs(chunk-post): add operator guide for fault-domain fan-out seeding
vilenarios Jul 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,8 @@ services:
- AR_IO_NODE_RELEASE=${AR_IO_NODE_RELEASE:-82-pre}
- CHUNK_POST_MIN_SUCCESS_COUNT=${CHUNK_POST_MIN_SUCCESS_COUNT:-}
- CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT=${CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT:-}
- CHUNK_POST_MIN_DISTINCT_DOMAINS=${CHUNK_POST_MIN_DISTINCT_DOMAINS:-}
- CHUNK_POST_PREFERRED_SOFT_FALLBACK=${CHUNK_POST_PREFERRED_SOFT_FALLBACK:-}
- CHUNK_POST_MAX_CONSECUTIVE_FAILURES=${CHUNK_POST_MAX_CONSECUTIVE_FAILURES:-}
- CHUNK_POST_SORTED_PEERS_CACHE_DURATION_MS=${CHUNK_POST_SORTED_PEERS_CACHE_DURATION_MS:-}
- CHUNK_POST_RESPONSE_TIMEOUT_MS=${CHUNK_POST_RESPONSE_TIMEOUT_MS:-}
Expand Down
2 changes: 2 additions & 0 deletions docs/envs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,8 @@ This document describes the environment variables that can be used to configure
| AWS_S3_CONTIGUOUS_DATA_PREFIX | String | undefined | Prefix for the S3 bucket to organize data |
| CHUNK_POST_MIN_SUCCESS_COUNT | String | "3" | Minimum count of 200 responses for of a given chunk to be considered properly seeded |
| CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT | String | "2" | Minimum count of 200 responses from preferred (tip) nodes for a chunk to be considered properly seeded. Set to 0 to disable preferred node requirement |
| CHUNK_POST_MIN_DISTINCT_DOMAINS | String | "0" | Minimum number of distinct fault domains (IP /24 for v4, /48 for v6) a chunk POST must land on among its successes. 0 = disabled (fault-domain-blind). Soft: degrades gracefully if the eligible peer set cannot supply this many distinct domains (never hard-fails on domain scarcity). Surfaced per-POST via the `X-AR-IO-Chunk-Placement-Domains` response header |
| CHUNK_POST_PREFERRED_SOFT_FALLBACK | Boolean | "false" | When true, a chunk POST may meet quorum via a strong distinct-domain quorum of discovered peers when NO preferred (tip) node was eligible (tips down/over-queue), removing the single-fault-domain tips SPOF. A `chunk_post_preferred_shortfall_total` metric is still emitted so a tips outage stays visible. Default false preserves the hard preferred-success requirement |
| CHUNK_POST_MAX_CONSECUTIVE_FAILURES | String | "5" | Maximum consecutive 4xx responses before stopping chunk broadcast. Only applies when no peers have accepted the chunk. Set to 0 to disable early termination |
| ARWEAVE_POST_DRY_RUN | Boolean | false | If true, simulates transaction header and chunk submission without posting to Arweave. `POST /tx` and `POST /chunk` return 200 OK as if successful; only the final network broadcast is skipped. Works on both port 3000 (Envoy) and port 4000 (direct). By default, transaction signatures and chunk merkle proofs are still validated before success. When disabled, Envoy routes these requests to trusted Arweave nodes instead; `GET /tx` is always proxied to the trusted node regardless of this setting. |
| ARWEAVE_POST_DRY_RUN_SKIP_VALIDATION | Boolean | false | If true (and `ARWEAVE_POST_DRY_RUN` is enabled), skips transaction signature verification and chunk merkle proof validation for faster testing. |
Expand Down
363 changes: 362 additions & 1 deletion monitoring/grafana/dashboards/default.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,9 @@
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": ["lastNotNull"],
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
Expand Down Expand Up @@ -1023,6 +1025,365 @@
],
"title": "I/O Utilization",
"type": "timeseries"
},
{
"id": 14,
"type": "row",
"title": "Chunk Fan-out Seeding (placement)",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 40
},
"panels": []
},
{
"id": 15,
"type": "timeseries",
"title": "Distinct fault domains per chunk POST",
"description": "Independent operators (IP /24·/48) a chunk landed on per POST. p10 is the seeding-risk tail: low p10 = a meaningful slice of chunks are landing on few fault domains.",
"datasource": "${datasource}",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 41
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "linear",
"lineWidth": 1,
"fillOpacity": 10,
"gradientMode": "none",
"spanNulls": false,
"showPoints": "auto",
"pointSize": 5,
"stacking": {
"group": "A",
"mode": "none"
},
"axisPlacement": "auto",
"axisColorMode": "text",
"scaleDistribution": {
"type": "linear"
},
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short"
},
"overrides": []
},
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": "${datasource}",
"editorMode": "code",
"expr": "sum(rate(chunk_post_distinct_domains_sum[$__rate_interval]))/sum(rate(chunk_post_distinct_domains_count[$__rate_interval]))",
"instant": false,
"legendFormat": "avg",
"range": true,
"refId": "A"
},
{
"datasource": "${datasource}",
"editorMode": "code",
"expr": "histogram_quantile(0.5, sum by (le) (rate(chunk_post_distinct_domains_bucket[$__rate_interval])))",
"instant": false,
"legendFormat": "p50",
"range": true,
"refId": "B"
},
{
"datasource": "${datasource}",
"editorMode": "code",
"expr": "histogram_quantile(0.1, sum by (le) (rate(chunk_post_distinct_domains_bucket[$__rate_interval])))",
"instant": false,
"legendFormat": "p10 (thinnest 10%)",
"range": true,
"refId": "C"
}
]
},
{
"id": 16,
"type": "timeseries",
"title": "Preferred (tip) shortfall rate",
"description": "POSTs where the preferred (tip) success count fell short. tips_unavailable = the tip /24 was down/over-queue (the SPOF the soft-preferred fallback covers); tips_failed = tips reachable but did not ack. Emitted even when the fallback still made the POST succeed.",
"datasource": "${datasource}",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 41
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "linear",
"lineWidth": 1,
"fillOpacity": 10,
"gradientMode": "none",
"spanNulls": false,
"showPoints": "auto",
"pointSize": 5,
"stacking": {
"group": "A",
"mode": "none"
},
"axisPlacement": "auto",
"axisColorMode": "text",
"scaleDistribution": {
"type": "linear"
},
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "ops"
},
"overrides": []
},
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": "${datasource}",
"editorMode": "code",
"expr": "sum(rate(chunk_post_preferred_shortfall_total[$__rate_interval])) by (reason)",
"instant": false,
"legendFormat": "{{reason}}",
"range": true,
"refId": "A"
}
]
},
{
"id": 17,
"type": "timeseries",
"title": "Distinct-domain target shortfall rate",
"description": "Only when CHUNK_POST_MIN_DISTINCT_DOMAINS>0. unmet = target achievable but not reached; degraded = eligible peer set could not supply the target (soft-capped, POST not failed on scarcity).",
"datasource": "${datasource}",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 49
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "linear",
"lineWidth": 1,
"fillOpacity": 10,
"gradientMode": "none",
"spanNulls": false,
"showPoints": "auto",
"pointSize": 5,
"stacking": {
"group": "A",
"mode": "none"
},
"axisPlacement": "auto",
"axisColorMode": "text",
"scaleDistribution": {
"type": "linear"
},
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "ops"
},
"overrides": []
},
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": "${datasource}",
"editorMode": "code",
"expr": "sum(rate(chunk_post_domain_shortfall_total[$__rate_interval])) by (reason)",
"instant": false,
"legendFormat": "{{reason}}",
"range": true,
"refId": "A"
}
]
},
{
"id": 18,
"type": "timeseries",
"title": "Chunk broadcast success rate",
"description": "Overall share of chunk broadcasts meeting the quorum verdict. Context for the diversity panels above.",
"datasource": "${datasource}",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 49
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "linear",
"lineWidth": 1,
"fillOpacity": 10,
"gradientMode": "none",
"spanNulls": false,
"showPoints": "auto",
"pointSize": 5,
"stacking": {
"group": "A",
"mode": "none"
},
"axisPlacement": "auto",
"axisColorMode": "text",
"scaleDistribution": {
"type": "linear"
},
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": "${datasource}",
"editorMode": "code",
"expr": "sum(rate(arweave_chunk_broadcast_total{status=\"success\"}[$__rate_interval]))/clamp_min(sum(rate(arweave_chunk_broadcast_total[$__rate_interval])),0.0001)",
"instant": false,
"legendFormat": "success ratio",
"range": true,
"refId": "A"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
}
]
}
],
"refresh": "auto",
Expand Down
Loading
Loading