feat: OTel integration in XRPLD#7770
Draft
pratikmankawde wants to merge 1417 commits into
Draft
Conversation
…ase9-metric-gap-fill # Conflicts: # docker/telemetry/docker-compose.yml
…ase10-workload-validation
- PerfLogImp::rpcEnd(): return after the requestId-not-found UNREACHABLE so a stale (now - epoch) duration is no longer recorded to the counter and histogram in release builds. - MetricsRegistry peer-version gauge: compare versions numerically via BuildInfo::encodeSoftwareVersion() instead of a lexicographic string compare, stripping the non-digit prefix so peer 'rippled-X.Y.Z' lines up with our bare 'X.Y.Z'. Fixes every peer counting as higher-version. - MetricsRegistry::stop(): call Shutdown() before ForceFlush() before reset() so the reader thread stops before teardown and no gauge callback fires during shutdown. - MetricsRegistry::start(): extract initExporterAndProvider() and initSyncInstruments() helpers to keep each function under the line limit; no behavior change. - time_in_current_state_seconds: read NetworkOPs::getServerStateDurationUs() (a lightweight accessor over StateAccounting) and convert microseconds to seconds, replacing the hardcoded 0.0.
The collector's log ingestion used a container path named after the old `rippled` binary and defaulted its host mount to a location that either needed root or diverged from where the telemetry configs actually write. Consolidate on one accessible log root: - container mount target renamed /var/log/rippled -> /var/log/xrpld - filelog glob -> /var/log/xrpld/*/debug.log - compose mount source defaults to the repo-relative ./data/logs (user-owned, no root), overridable via XRPLD_LOG_DIR - devnet telemetry cfg writes to data/logs/devnet/debug.log so it lands one subdir below the mount root and matches the glob - integration-test.sh sets XRPLD_LOG_DIR to its own workdir - docs/runbook/task-list updated to match The default xrpld config (cfg/xrpld-example.cfg) is intentionally left untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ase9-metric-gap-fill # Conflicts: # docker/telemetry/docker-compose.yml # docker/telemetry/otel-collector-config.yaml # docker/telemetry/xrpld-telemetry.cfg
…ase10-workload-validation
Extend the log-mount consolidation to the phase-10 workload-validation assets, which still used the old `rippled` container path: - docker-compose.workload.yaml: mount target renamed to /var/log/xrpld, source made overridable via XRPLD_LOG_DIR (default /tmp/xrpld-validation) - otel-collector-config.grafanacloud.yaml: single glob /var/log/xrpld/*/debug.log - run-full-validation.sh sets XRPLD_LOG_DIR to its workdir Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
xrpld resolves a relative [debug_logfile] against the config file's own directory (docker/telemetry), not the working directory. The value `docker/telemetry/data/logs/devnet/debug.log` therefore doubled the prefix to docker/telemetry/docker/telemetry/... — the collector's mount never saw the log, so no lines reached Loki. Use the config-dir-relative form `data/logs/devnet/debug.log`, which resolves to docker/telemetry/data/logs/devnet/debug.log — exactly the dir the compose stack bind-mounts as /var/log/xrpld. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ase9-metric-gap-fill # Conflicts: # docker/telemetry/xrpld-telemetry.cfg
…ase10-workload-validation
…tion The workload validator queried each expected metric once, immediately after a fixed post-workload propagation wait. Several beast::insight metrics (ledger-age and peer-finder gauges, overlay-traffic and rpc-request counters) only populate after the node validates ledgers and sustains peer traffic, then travel a 1s OTLP export + 15s Prometheus scrape before they are queryable. On a slower CI runner that pipeline can settle after the wait ends, so the single query raced and reported "0 series", failing 12 checks that pass locally with the same config and binary. Poll each metric on the /api/v1/series endpoint until it appears or a 45s window (two scrape cycles) elapses. Present metrics still return on the first query with no added delay; a genuinely-absent metric still fails after the timeout. Makes the check robust to runner speed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tance exported_instance is a Prometheus scrape artifact that only exists when Prometheus scrapes a target with a conflicting instance label. Users running push-based pipelines (remote_write, OTLP) never get this label. service_instance_id is always present via resource_to_telemetry_conversion regardless of ingestion path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tance exported_instance is a Prometheus scrape artifact that only exists when Prometheus scrapes a target with a conflicting instance label. Users running push-based pipelines (remote_write, OTLP) never get this label. service_instance_id is always present via resource_to_telemetry_conversion regardless of ingestion path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…se8-log-correlation
…tance Merge forward from phase6/7 and apply the same fix to phase9's own dashboards and alerting rules. exported_instance is a Prometheus scrape artifact absent in push-based pipelines; service_instance_id is always present via resource_to_telemetry_conversion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ase10-workload-validation
…d docs Update the transform/cloudlabels processor to set service_instance_id instead of exported_instance, matching the local Prometheus path where resource_to_telemetry_conversion promotes service.instance.id with that exact label name. Also update source comments and documentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…se8-log-correlation
…ase9-metric-gap-fill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ase10-workload-validation
Add missing <cstdint> include for std::uint32_t in Telemetry.h. Add braces around single-line if bodies in Telemetry.cpp. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…hase1c-rpc-integration
…hase2-rpc-tracing
Replace std::transform with std::ranges::transform in Redaction.cpp. Replace std::all_of with std::ranges::all_of in Redaction test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unused SpanNames.h from Transactor.cpp (transitively included via TxApplySpanNames.h). Remove unused <ranges> from TraceContextValidation.h (std::ranges::any_of lives in <algorithm>). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…consensus-tracing
…ase9-metric-gap-fill
…se8-log-correlation
…ase10-workload-validation
The docker/telemetry/alloy/config.alloy node config feeds Grafana Cloud via the OTLP gateway (which promotes resource attributes to labels server-side). Add the resource_group / iac_group build to its metric resource statements so nodes using this Alloy path get the same grouped legend labels the dashboards expect. This is a generic node config (no perf-iac attrs), so iac_group's guard never fires here and only resource_group is built; the iac_group statement is kept for parity and activates automatically on a perf-comparison node. Guards check != "" as well as != nil since OTTL Concat emits a stray separator for an empty element. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the Alloy config the spanmetrics connector derives span_calls_total / span_duration_* from the tagged TRACES and sends them straight to the batch processor, bypassing the metric resource statements. The grouped legend labels were built only in metric_statements, so span-derived RED metrics carried no resource_group / iac_group and span-based dashboard panels rendered empty group legends. Build the same labels in trace_statements too (mirroring how the tier attrs are already set in both), so spans carry them before spanmetrics derives from them. The reference otel collector is unaffected: there spanmetrics is a connector feeding the metrics pipeline, which runs transform/legendgroups on all inputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Panels that group by a distinguishing dimension (consensus_mode, tx_type,
command, method, stage, ...) already fold that dimension's value into the
curated series name via label_replace(..., "series", "<prefix> $1", "<dim>",
"(.*)"). The migration also added a separate [${__field.labels.<dim>}] bracket,
so those legends rendered the value twice, e.g. "proposing [proposing] [...]"
on Consensus Mode Over Time.
Remove the dimension bracket from all 29 affected panel Display names, leaving
${__field.labels.series} [resource_group] [iac_group]; the series name already
conveys the dimension. The migration script no longer emits the bracket either.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ase10-workload-validation
The Display name templated the brackets: "${series} [${resource_group}]
[${iac_group}]". When a group label was absent (e.g. iac_group on a non-perf
node) the "${}" rendered empty, leaving a literal "[]" in the legend. The
empty-bracket cleanup renameByRegex could not remove it: Grafana runs that
transform on the pre-interpolation Display name template, which contains no
"[]" -- the "[]" only appears after variable interpolation, at render time.
Fix it at the source: the collector now bakes the brackets and comma separators
into the group-label VALUES (e.g. resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]"). The Display name becomes bracket-free --
"${series} ${resource_group} ${iac_group}" -- so an absent group is simply an
absent label and renders as nothing. Also switches the intra-group separator
from "-" to ", " (e.g. "[node, mainnet]"). Drop the now-dead cleanup transform
from every panel; value-mapping renameByRegex rules are preserved.
Collector-side label building lives in the paired commits on the collector
config branches. This commit is the dashboard half for the eleven dashboards
this branch owns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Display name templated the brackets: "${series} [${resource_group}]
[${iac_group}]". When a group label was absent the "${}" rendered empty,
leaving a literal "[]" in the legend, and the cleanup renameByRegex could not
remove it (Grafana runs that transform on the pre-interpolation template, which
has no "[]" -- it only appears after interpolation at render time).
Bake the brackets and comma separators into the group-label VALUES in the
collector instead (resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]"), so the Display name is bracket-free
("${series} ${resource_group} ${iac_group}") and an absent group renders as
nothing. Separator changes from "-" to ", ". Drop the dead cleanup transform.
Covers this branch's local collector config and its three dashboards
(ledger-data-sync, network-traffic, overlay-traffic-detail).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the local collector change on the Grafana Cloud collector config and the in-repo Alloy config: build the group-label VALUES with brackets and comma separators (resource_group="[node, mainnet]", iac_group="[branch, role, work-item]") in both trace and metric resource statements, so span-derived and native metrics carry the same bracketed values. This lets the dashboards use a bracket-free Display name and avoids the empty "[]" that a templated bracket would leave when a group is absent (Grafana's renameByRegex runs pre-interpolation and cannot strip it). Separator changes from "-" to ", ". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ase9-metric-gap-fill
…se8-log-correlation
…ase10-workload-validation # Conflicts: # docker/telemetry/grafana/dashboards/ledger-data-sync.json
Update the config.alloy metric_statements comment to describe the current
bracket-baked, ", "-joined group-label values ("[node, mainnet]"), matching the
implementation. Comment-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…up labels
The collector-built resource_group / iac_group labels did not survive the
dashboards' own PromQL aggregation: panels that sum by(service_instance_id,
xrpl_branch, xrpl_node_role, ...) drop any label not named in the by() clause,
so the derived group labels were aggregated away and the Display name rendered
empty ("Observing" with no node). Building them in the collector also coupled
every dashboard to a lockstep collector redeploy on every node.
Switch the legend Display name to reference the raw labels the panels already
group by, which are always present by construction and need no collector
support:
${__field.labels.series} ${__field.labels.service_instance_id}
${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role}
${__field.labels.xrpl_work_item}
${__field.labels.X} renders empty for a label a series lacks (verified in
Grafana), so dev/mainnet nodes show just "<name> <node>" and perf nodes add the
branch/role/work-item, with no empty brackets and no double spaces. Verified
live across timeseries, stat, bargauge, piechart, gauge and cross-node
aggregation panels.
Revert the transform/legendgroups processor from the local collector config;
this file is now identical to its pre-change state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…up labels
The collector-built resource_group / iac_group labels did not survive the
dashboards' own PromQL aggregation: panels that sum by(service_instance_id,
xrpl_branch, xrpl_node_role, ...) drop any label not in the by() clause, so the
derived group labels were aggregated away and the Display name rendered empty.
Switch the legend Display name to the raw labels the panels already group by,
which are always present by construction:
${__field.labels.series} ${__field.labels.service_instance_id}
${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role}
${__field.labels.xrpl_work_item}
Absent labels render empty (no brackets, no gaps). Covers the eleven dashboards
this branch owns. The matching collector-config revert lands on the branches
that introduced those configs (phase-7 local, phase-10 cloud/alloy).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…figs) The legend now uses raw labels in the dashboard Display name (see the paired dashboard commit), so the collector no longer needs to build resource_group / iac_group. Those derived labels also could not survive the dashboards' sum by() aggregation, which is why the grouped approach was dropped. Revert transform/legendgroups and its transform/cloudlabels datapoint copies from the Grafana Cloud collector config and the in-repo Alloy config; both files are now identical to their pre-change state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…se8-log-correlation # Conflicts: # docker/telemetry/otel-collector-config.yaml
…ase9-metric-gap-fill
…ase10-workload-validation
Group the node + perf-iac identity labels inside one pair of brackets in the
legend Display name:
${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]
so a series reads e.g. "Queue Depth [validator-0 test:pr:abc123 validator
RIPD-7455]" on a perf node and "Queue Depth [aws-dev-xrpl-1]" on a plain node.
Absent perf-iac labels render empty and collapse to a single space before the
closing bracket (HTML/SVG whitespace collapse) — the only cosmetic artifact on
non-perf nodes; no empty "[]" and no data impact. Verified on a live local
Grafana render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Group the node + perf-iac identity labels inside one pair of brackets in the legend Display name so a series reads e.g. "Queue Depth [validator-0 test:pr:abc123 validator RIPD-7455]" on a perf node and "Queue Depth [aws-dev-xrpl-1]" on a plain node. Absent perf-iac labels collapse to a single space before the closing bracket; no empty "[]", no data impact. Covers the eleven dashboards this branch owns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…se8-log-correlation
…ase9-metric-gap-fill
…ase10-workload-validation
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
…escriptions Rework the legend identity bracket across all 14 dashboards so it reads "<series> [instance, branch, work_item]" (comma-separated, Node Role dropped) and collapses cleanly when perf-iac labels are absent -- "<series> [instance]" with no empty "[, ]" gaps. - Build one xrpl_ident label per query via label_join + a single label_replace that trims leading/trailing empty segments and brackets the result. Done at the outermost (post-aggregation) level so it survives each panel's by() clause. - Add service_instance_id to the by() of 6 panels that grouped only by node_role, so same-branch validator/peer series stay distinguishable after dropping the node_role legend token. - Fold the two legendFormat two-bracket panels (Transaction Results by Type, Tx Apply Pipeline Latency by Type and Stage) into the same single-bracket form. - Rename the annotation query label "Perf runs (perf-iac)" to "Annotate perf-iac runs". - Revert dashboard-level descriptions to plain text (the folder-list info tooltip escapes HTML, so <br>/markdown render literally). Verified: output byte-identical to the prior multi-call cleanup over a 2-day live perf window; all 14 files valid JSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regroup all 47 panels under correct subsystem rows and fix the broken row structure. Previously the "NodeStore I/O" row had no gridPos and collided at y=32 with an appended "Extended Metrics" catch-all row, so NodeStore I/O rendered empty and its panels landed under the wrong header. - Rows: Overview (top), Operating Mode, NodeStore I/O, Jobs, Caches, Server Info, Complete Ledgers & DB, Ledger Economy. Drop the stray "Extended Metrics" row; each panel now sits under its subsystem. - Normalize panel heights to three buckets: 12 (standard), 16 (emphasis / detail plots), 24 (Operating Mode state timeline). Widths stay half (12) or full (24). - Contiguous, non-overlapping gridPos; all 47 panels preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ions Sync the Node Health dashboard with panels and field settings refined in Grafana, and document the ledger-convergence panels: - Add descriptions for "Validated Ledger Seq — Current (Stat)" (per-node lag behind the network tip) and "Validated Ledger Seq — Convergence (Max − Min)" (seq spread across nodes), in the standard 7-field panel format. - Peer Count: threshold colours (red < 5, yellow < 10, green >= 10) so a low peer count reads red rather than using a value gradient. - Minor field-config fixes on Last Close and NuDB Storage panels (axis label, threshold cleanup) as adjusted in Grafana. Add the Grafana duration unit "dthms" to the cspell dictionary. Panel content stays at parity with the Grafana Cloud copy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
High Level Overview of Change
Adds full OpenTelemetry (OTel) integration to
xrpld: distributed traces, native OTel metrics, and log/trace correlation, with a companion observability stack (Collector, Tempo, Prometheus, Loki, Grafana) and a workload harness that validates the end-to-end signal path.Grafana Dashboards
Scope:
traceparentpropagation across RPC and peer boundaries.xrpldlogs so Loki entries deep-link into Tempo.Context of Change
xrpldhad only StatsD counters and free-form logs — no distributed traces, no structured attributes, and no way to correlate a slow RPC to the consensus round it landed in. This change introduces an OTel-native pipeline so operators can trace requests end-to-end, alert on structured SLIs, and jump from a log line to its trace.Delivered in ten internal phases (plan docs under
OpenTelemetryPlan/), landed here as a single feature branch offdevelop. Telemetry is off by default; enable via[telemetry]/[insight] server=otelin the config.API Impact
libxrplchange — adds telemetry primitives (SpanGuard,SpanNames, metrics registry) usable bylibxrplconsumers; existing symbols unchanged.traceparentrides existing peer message headers and is optional.No wire-format or JSON-RPC schema changes. Config gains an opt-in
[telemetry]stanza and an[insight] server=otelvalue.Test Plan
docker/telemetry/workload compose brings up 5 validators + full OTel stack;make telemetry-workload-upruns the RPC/TX load generators.