Document shipped worker-wide Datadog and Sentry observability for the analyzer worker - #885
Document shipped worker-wide Datadog and Sentry observability for the analyzer worker#885promptless[bot] wants to merge 2 commits into
Conversation
…ervability Fill in the worker-wide observability content on the Observability and Configuration reference pages, replacing the 'forthcoming' placeholders now that Promptless/promptless#4694 has shipped worker-wide Datadog tracing, structured JSON logging, and Sentry error reporting (Sentry DSN sourcing per the follow-up #4699). Grounded in merged source.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![promptless[bot]](https://avatars.githubusercontent.com/in/979718?s=60&v=4){kind=small}
| ## Worker-wide Datadog tracing | ||
|
|
||
| First-class, worker-wide observability — starting the whole worker under Datadog tracing and reporting errors to Sentry — is in progress and not yet shipped. When it lands, this page documents the configuration it introduces. Until then, configure only the analysis tracing described above; there is no supported worker-wide Datadog or Sentry configuration to set yet. | ||
| The whole worker process starts under `ddtrace-run`, the container entrypoint, so Datadog APM traces cover the worker's request handling — not only the analysis child process. Tracing is opt-in: it stays disabled until you set `DD_TRACE_ENABLED=true`, and the image ships with it off. |
There was a problem hiding this comment.
Dockerfile sets ENV DD_TRACE_ENABLED=false (L21) and ENTRYPOINT ["ddtrace-run", "instruction-hub-worker"] (L35), confirming the whole worker process starts under ddtrace-run with tracing disabled by default.
| First-class, worker-wide observability — starting the whole worker under Datadog tracing and reporting errors to Sentry — is in progress and not yet shipped. When it lands, this page documents the configuration it introduces. Until then, configure only the analysis tracing described above; there is no supported worker-wide Datadog or Sentry configuration to set yet. | ||
| The whole worker process starts under `ddtrace-run`, the container entrypoint, so Datadog APM traces cover the worker's request handling — not only the analysis child process. Tracing is opt-in: it stays disabled until you set `DD_TRACE_ENABLED=true`, and the image ships with it off. | ||
|
|
||
| Turn it on through the Helm chart's Datadog values (the `observability.datadog.enabled` value). Enabling it renders `DD_TRACE_ENABLED=true` and the `DD_*` connection variables for you — you configure the agent's site and URL through the chart's Datadog values rather than setting `DD_TRACE_AGENT_URL` and `DD_SITE` by hand — and applies Datadog Unified Service Tagging labels and log-collection annotations to both the worker Deployment and the migration Job. Those annotations also mean the worker's stdout JSON logs are collected by the Datadog agent, not only its traces. If you deploy without the chart, set these variables directly and run a reachable Datadog agent for the telemetry to land. |
There was a problem hiding this comment.
datadogLabels and observabilityEnv named templates render Datadog Unified Service Tagging labels and DD_* env vars only when .Values.observability.datadog.enabled (Sentry env gated on .Values.observability.sentry.enabled).
| First-class, worker-wide observability — starting the whole worker under Datadog tracing and reporting errors to Sentry — is in progress and not yet shipped. When it lands, this page documents the configuration it introduces. Until then, configure only the analysis tracing described above; there is no supported worker-wide Datadog or Sentry configuration to set yet. | ||
| The whole worker process starts under `ddtrace-run`, the container entrypoint, so Datadog APM traces cover the worker's request handling — not only the analysis child process. Tracing is opt-in: it stays disabled until you set `DD_TRACE_ENABLED=true`, and the image ships with it off. | ||
|
|
||
| Turn it on through the Helm chart's Datadog values (the `observability.datadog.enabled` value). Enabling it renders `DD_TRACE_ENABLED=true` and the `DD_*` connection variables for you — you configure the agent's site and URL through the chart's Datadog values rather than setting `DD_TRACE_AGENT_URL` and `DD_SITE` by hand — and applies Datadog Unified Service Tagging labels and log-collection annotations to both the worker Deployment and the migration Job. Those annotations also mean the worker's stdout JSON logs are collected by the Datadog agent, not only its traces. If you deploy without the chart, set these variables directly and run a reachable Datadog agent for the telemetry to land. |
There was a problem hiding this comment.
Worker Deployment includes datadogLabels (L7, L24), the ad.datadoghq.com/worker.logs log-collection annotation (L28), and observabilityEnv (L66, not shown in this range) — confirms Datadog wiring applies to the Deployment.
| First-class, worker-wide observability — starting the whole worker under Datadog tracing and reporting errors to Sentry — is in progress and not yet shipped. When it lands, this page documents the configuration it introduces. Until then, configure only the analysis tracing described above; there is no supported worker-wide Datadog or Sentry configuration to set yet. | ||
| The whole worker process starts under `ddtrace-run`, the container entrypoint, so Datadog APM traces cover the worker's request handling — not only the analysis child process. Tracing is opt-in: it stays disabled until you set `DD_TRACE_ENABLED=true`, and the image ships with it off. | ||
|
|
||
| Turn it on through the Helm chart's Datadog values (the `observability.datadog.enabled` value). Enabling it renders `DD_TRACE_ENABLED=true` and the `DD_*` connection variables for you — you configure the agent's site and URL through the chart's Datadog values rather than setting `DD_TRACE_AGENT_URL` and `DD_SITE` by hand — and applies Datadog Unified Service Tagging labels and log-collection annotations to both the worker Deployment and the migration Job. Those annotations also mean the worker's stdout JSON logs are collected by the Datadog agent, not only its traces. If you deploy without the chart, set these variables directly and run a reachable Datadog agent for the telemetry to land. |
There was a problem hiding this comment.
Migration Job includes datadogLabels (L8, L19), the ad.datadoghq.com/migrate.logs annotation (L22), and observabilityEnv (L51, not shown in this range) — confirms the same Datadog wiring also applies to the migration Job, not only the worker Deployment.
|
|
||
| ## Structured logging | ||
|
|
||
| The worker emits every log line as JSON to stdout, one object per line. This emission is always on and not tied to Datadog or Sentry, so any log pipeline can parse the output whether or not either is enabled. The logs go to stdout regardless; when you enable Datadog, its log-collection annotations forward that stdout to the Datadog agent. |
There was a problem hiding this comment.
configure_logging(settings.log_level) is called unconditionally before configure_datadog()/configure_sentry(), which each internally no-op when disabled — confirms structured JSON logging is always on and independent of Datadog/Sentry.
|
|
||
| The worker emits every log line as JSON to stdout, one object per line. This emission is always on and not tied to Datadog or Sentry, so any log pipeline can parse the output whether or not either is enabled. The logs go to stdout regardless; when you enable Datadog, its log-collection annotations forward that stdout to the Datadog agent. | ||
|
|
||
| Each record carries a UTC ISO-8601 `timestamp`, the level as `status`, the `logger.name`, the `message`, any extra fields, and `error.stack` when an exception is present. Verbosity is controlled by `INSTRUCTION_HUB_LOG_LEVEL`, which defaults to `INFO`; see [Configuration reference](/docs/governance/deploy-the-worker/configuration-reference). |
There was a problem hiding this comment.
_JsonLogFormatter builds each record's timestamp (UTC ISO-8601, milliseconds), status (level), logger.name, message, extra fields, and error.stack when exc_info/stack_info is present.
|
|
||
| Each record carries a UTC ISO-8601 `timestamp`, the level as `status`, the `logger.name`, the `message`, any extra fields, and `error.stack` when an exception is present. Verbosity is controlled by `INSTRUCTION_HUB_LOG_LEVEL`, which defaults to `INFO`; see [Configuration reference](/docs/governance/deploy-the-worker/configuration-reference). | ||
|
|
||
| When Datadog log injection is on (`DD_LOGS_INJECTION=true`), each record also gains the trace-correlation fields `dd.trace_id` and `dd.span_id` — opaque identifiers that link the logs back to their traces. |
There was a problem hiding this comment.
Confirms DD_LOGS_INJECTION (default true in ddtrace) is the toggle that makes the tracer inject trace context into logs when ddtrace-run/ddtrace.auto is used; the chart hardcodes DD_LOGS_INJECTION="true" only when Datadog is enabled (_helpers.tpl L53-54 at commit eb66ac3).
Source: https://ddtrace.readthedocs.io/en/stable/configuration.html
|
|
||
| ## Sentry error reporting | ||
|
|
||
| Sentry error reporting is opt-in: it stays off until you set a Sentry DSN (`SENTRY_DSN`). |
There was a problem hiding this comment.
configure_sentry() returns False without calling sentry_sdk.init when settings.sentry_dsn is None — confirms Sentry is off unless SENTRY_DSN is set.
|
|
||
| Sentry error reporting is opt-in: it stays off until you set a Sentry DSN (`SENTRY_DSN`). | ||
|
|
||
| When enabled, the worker reports uncaught exceptions from its HTTP layer (FastAPI and Starlette) and turns ERROR-level log records into Sentry events; lower-level records are attached as breadcrumbs. Sentry performance tracing stays off; Datadog owns application tracing, so the worker never sends Sentry APM data. The worker initializes Sentry with personally identifiable information disabled, so it does not attach cookies, the client's IP address, or logged-in user identity (username, ID, email) to events. Events are tagged with the service and the deployment name, and carry the deployment instance as the event's server name, so you can tell which worker deployment produced them. |
There was a problem hiding this comment.
sentry_sdk.init uses StarletteIntegration/FastApiIntegration (uncaught exceptions) plus LoggingIntegration(level=INFO, event_level=ERROR) (ERROR->events, lower->breadcrumbs); no traces_sample_rate/traces_sampler is set; send_default_pii=False; set_tag("service", ...) and set_tag("deployment_name", ...) plus server_name=deployment_instance_id.
|
|
||
| When enabled, the worker reports uncaught exceptions from its HTTP layer (FastAPI and Starlette) and turns ERROR-level log records into Sentry events; lower-level records are attached as breadcrumbs. Sentry performance tracing stays off; Datadog owns application tracing, so the worker never sends Sentry APM data. The worker initializes Sentry with personally identifiable information disabled, so it does not attach cookies, the client's IP address, or logged-in user identity (username, ID, email) to events. Events are tagged with the service and the deployment name, and carry the deployment instance as the event's server name, so you can tell which worker deployment produced them. | ||
|
|
||
| Use a Sentry project dedicated to the worker, separate from your other alert streams. Keeping the worker's exception data in its own project means worker errors — and whoever can see them — stay separate from your runtime and eval alert streams. Supply the DSN as a secret through your deployment's secret store — with the Helm chart, provide it through the chart's Sentry values rather than as a plain value — so it is never committed in plaintext. |
There was a problem hiding this comment.
SENTRY_DSN env is sourced via secretKeyRef built from observability.sentry.existingSecretName/dsnKey (the chart's Sentry values) rather than a plain value — confirms the DSN is supplied as a secret through the chart's Sentry configuration, without asserting which secret/key name a given deployment uses (that is set per values file, e.g. values-prod.yaml).
| Use a Sentry project dedicated to the worker, separate from your other alert streams. Keeping the worker's exception data in its own project means worker errors — and whoever can see them — stay separate from your runtime and eval alert streams. Supply the DSN as a secret through your deployment's secret store — with the Helm chart, provide it through the chart's Sentry values rather than as a plain value — so it is never committed in plaintext. | ||
|
|
||
| <Aside type="note"> | ||
| Promptless's own production deploy enforces this isolation. Before it upgrades the worker, the deploy verifies that the worker's secret actually holds the dedicated Sentry DSN and fails the deploy if that key is missing or empty. Your own `helm upgrade` does not run that workflow, but the same rule applies: keep the worker's DSN in its own secret. |
There was a problem hiding this comment.
PR #4699 repointed the "Verify worker secrets exist" step: it now also fetches the runtime-env secret and requires runtime-env["INSTRUCTION_HUB_WORKER_SENTRY_DSN"] to be present and non-empty (in addition to instruction-hub-worker-prod's own required keys, which no longer include sentry-dsn), raising SystemExit if any required key across either secret is missing/empty. This step still runs before the "Deploy worker to EKS" helm upgrade step, so the doc's unnamed-key description ("verifies the worker's secret actually holds the dedicated Sentry DSN and fails the deploy if that key is missing or empty") remains accurate.
|
|
||
| ## Observability variables | ||
|
|
||
| These variables are optional. They configure worker-wide Datadog tracing, structured logging, and Sentry error reporting. Datadog and Sentry are both off unless you turn them on. When you deploy with the Helm chart, its Datadog values render the `DD_*` variables and its Sentry values supply `SENTRY_DSN` and the related Sentry settings. For how each layer behaves, see [Observability](/docs/governance/deploy-the-worker/observability). The `DD_TRACE_AGENT_URL` here points the worker-wide tracer at your Datadog agent; the analysis child process connects through `DD_AGENT_HOST` and the other `DD_TRACE_*` variables shown in the [Analysis tracing](/docs/governance/deploy-the-worker/observability) section instead. |
There was a problem hiding this comment.
Default chart values.yaml sets observability.datadog.enabled=false and observability.sentry.enabled=false — confirms both are off by default and rendered only when the chart's values enable them.
| | `DD_SERVICE` | The Datadog service name for the worker's traces, logs, and metrics. | | ||
| | `DD_ENV` | The environment tag applied to the worker's Datadog telemetry. | | ||
| | `DD_VERSION` | The version tag for Datadog telemetry; set to the worker image tag. | | ||
| | `DD_SITE` | The Datadog site the agent reports to, for example `datadoghq.com`. | |
There was a problem hiding this comment.
Chart default observability.datadog.site is "datadoghq.com" — a generic example, correctly distinct from the Promptless-prod-only "us5.datadoghq.com" override in values-prod.yaml, which the doc does not present as a customer requirement.
| | `DD_TRACE_AGENT_URL` | The URL of the Datadog agent that receives the worker's traces. | | ||
| | `DD_LOGS_INJECTION` | Adds the trace-correlation fields `dd.trace_id` and `dd.span_id` to the JSON logs so logs link to traces. | | ||
| | `DD_TRACE_ANALYTICS_ENABLED` | Enables Datadog trace analytics. | | ||
| | `INSTRUCTION_HUB_LOG_LEVEL` | The minimum level the worker logs. Logs are always JSON; this sets verbosity. Defaults to `INFO`. | |
There was a problem hiding this comment.
WorkerSettings.log_level defaults to WorkerLogLevel.INFO.
| | `DD_TRACE_ANALYTICS_ENABLED` | Enables Datadog trace analytics. | | ||
| | `INSTRUCTION_HUB_LOG_LEVEL` | The minimum level the worker logs. Logs are always JSON; this sets verbosity. Defaults to `INFO`. | | ||
| | `SENTRY_DSN` | The Sentry DSN for the worker's dedicated Sentry project. Sentry is off unless this is set. | | ||
| | `SENTRY_ENVIRONMENT` | The environment tag on Sentry events. Falls back to the deployment environment, then `local`. | |
There was a problem hiding this comment.
sentry_environment=_optional(source, "SENTRY_ENVIRONMENT") or _optional(source, "ENV") or "local" — confirms the SENTRY_ENVIRONMENT -> ENV -> "local" fallback chain.
| | `INSTRUCTION_HUB_LOG_LEVEL` | The minimum level the worker logs. Logs are always JSON; this sets verbosity. Defaults to `INFO`. | | ||
| | `SENTRY_DSN` | The Sentry DSN for the worker's dedicated Sentry project. Sentry is off unless this is set. | | ||
| | `SENTRY_ENVIRONMENT` | The environment tag on Sentry events. Falls back to the deployment environment, then `local`. | | ||
| | `SENTRY_RELEASE` | The release tag on Sentry events; set to the worker image tag. | |
There was a problem hiding this comment.
SENTRY_RELEASE value is set from .Values.image.tag (falling back to .Chart.AppVersion) — confirms SENTRY_RELEASE is set to the worker image tag.
|
|
||
| Turn it on through the Helm chart's Datadog values (the `observability.datadog.enabled` value). Enabling it renders `DD_TRACE_ENABLED=true` and the `DD_*` connection variables for you — you configure the agent's site and URL through the chart's Datadog values rather than setting `DD_TRACE_AGENT_URL` and `DD_SITE` by hand — and applies Datadog Unified Service Tagging labels and log-collection annotations to both the worker Deployment and the migration Job. Those annotations also mean the worker's stdout JSON logs are collected by the Datadog agent, not only its traces. If you deploy without the chart, set these variables directly and run a reachable Datadog agent for the telemetry to land. | ||
|
|
||
| The worker-wide tracer connects to the agent through `DD_TRACE_AGENT_URL`; the analysis child process connects through `DD_AGENT_HOST` (see [Analysis tracing](#analysis-tracing) above), and setting one does not configure the other. |
There was a problem hiding this comment.
ddtrace configuration reference documents DD_TRACE_AGENT_URL and DD_AGENT_HOST as separate variables (DD_AGENT_HOST sets only the agent hostname; DD_TRACE_AGENT_URL sets the full agent URL and overrides/ignores DD_AGENT_HOST only if both are set in the same process) — confirms they are independent knobs and setting one does not populate or configure the other across the worker-wide process and the separately-launched analysis child process.
Source: https://ddtrace.readthedocs.io/en/stable/configuration.html
|
|
||
| ## Observability variables | ||
|
|
||
| These variables are optional. They configure worker-wide Datadog tracing, structured logging, and Sentry error reporting. Datadog and Sentry are both off unless you turn them on. When you deploy with the Helm chart, its Datadog values render the `DD_*` variables and its Sentry values supply `SENTRY_DSN` and the related Sentry settings. For how each layer behaves, see [Observability](/docs/governance/deploy-the-worker/observability). The `DD_TRACE_AGENT_URL` here points the worker-wide tracer at your Datadog agent; the analysis child process connects through `DD_AGENT_HOST` and the other `DD_TRACE_*` variables shown in the [Analysis tracing](/docs/governance/deploy-the-worker/observability) section instead. |
There was a problem hiding this comment.
Same ddtrace configuration reference as cited on the Observability page — DD_TRACE_AGENT_URL (worker-wide) and DD_AGENT_HOST (analysis child, per the unchanged Analysis tracing table) are independent environment variables; confirms this table's cross-reference sentence.
Source: https://ddtrace.readthedocs.io/en/stable/configuration.html
1 participant
Open in Promptless
From: @adit-chandra (source PR Promptless/promptless#4694)
Fills in the worker-wide observability content that the governance docs spine (PR #884, now merged) deliberately left marked "forthcoming" because it was gated on an open source PR. That source work has now shipped, so this replaces the placeholders on two already-published governance pages with the real, shipped configuration.
deploy-the-worker/observability.mdx): replaces the "Forthcoming: worker-wide Datadog and Sentry" placeholder (and the now-outdated intro and Analysis-tracing note) with three shipped sections — worker-wide Datadog tracing (the whole worker runs underddtrace-run; opt-in and off unless enabled; enabling the chart's Datadog values also collects the worker's JSON logs), always-on structured JSON logging (field names and Datadog trace-correlation fields), and Sentry error reporting (opt-in viaSENTRY_DSN; captures uncaught FastAPI/Starlette exceptions and ERROR logs; performance tracing disabled because Datadog owns application tracing; a dedicated worker Sentry project kept separate from other alert streams; and Promptless's production deploy preflight that fails when the worker DSN is unprovisioned).deploy-the-worker/configuration-reference.mdx): adds an "Observability variables" section documenting theDD_*,INSTRUCTION_HUB_LOG_LEVEL, andSENTRY_*variables, with a secret-handling caution. Required and analysis variable sections are unchanged.Grounded in the merged source. The Sentry DSN-sourcing detail is written at a durable, customer-facing level (secret store / chart values) so it stays accurate after the follow-up PR #4699 (merged) that changed how the production DSN is projected into the managed secret. Navigation was already wired by the spine, so no
astro.config.mjschanges were needed.Trigger Events