[feat] Extend events beyond deployments

[junaway]

What this unlocks

This PR turns the events pipeline from "one event type per deployment commit" into a general read/commit/log surface across the major Git-style entities, with a real quota meter and a real feature flag wired all the way through.

Twenty-nine new event types ship in EventType and (subscribably) in WebhookEventType: trace reads (traces.fetched, traces.queried), testcase reads (testcases.fetched, testcases.queried), and the full retrieved / fetched / queried / logged / committed lifecycle for applications, queries, testsets, evaluators, and environments at the revision level. Every type is webhook-subscribable. Webhook subscribers and the POST /events/query audit log now see real-user activity, not just the pre-existing environments.revisions.committed deploy precedent.

Counter.EVENTS_INGESTED is now a real, enforced meter — same two-layer pattern as Counter.TRACES_INGESTED. L1 lives inside core/events/utils.py::_safe_publish: a cache=True soft check that silently drops the publish when the org is over quota (read/commit responses are never 429'd by event metering). L2 lives in EventsWorker.process_batch: an authoritative adjust() per org with the full per-org delta in the batch, mirroring the tracing worker. Plans declare the quota inline (Quota(retention=..., period=Period.MONTHLY) for Hobby/Pro/Business, unlimited for Agenta and Self-hosted Enterprise) and operators can flip a limit=... per plan without any code change.

Flag.AUDIT is a new entitlement flag that gates POST /events/query — the human-facing audit-log query surface. Hobby and Pro return NOT_ENTITLED_RESPONSE(Tracker.FLAGS) (Business / Agenta / Self-hosted are entitled). Ingest and webhook delivery are not gated by Flag.AUDIT — events keep flowing into the events table and to webhook subscribers regardless of audit-log entitlement, so an upgrade makes historical events queryable immediately and webhook integrations work for every paid plan. The flag is in CONSTRAINTS[BLOCKED] so orgs can't promote themselves.

The new helper layer in core/events/utils.py enforces the read=router / write=service split: read events emit at the router boundary after the response is materialized (so internal service-to-service fetches that resolve refs or hydrate diff state stay silent), and commit events emit from inside commit_*_revision(...) at the service layer (so direct commit routes, simple-service create/edit, deploy paths, fork, and defaults seeding all produce exactly one event without double-publishing). The split is documented in the module docstring with the specific double-counting cases it prevents (commit_environment_revision calling query_environment_revisions to compute the diff; TracingService.query_traces calling queries_service.fetch_query_revision to resolve the saved query; evaluation runs fetching hundreds of revisions per scenario).

The publisher side now reads scope from the ambient AuthScope (ContextVar set by the auth middleware) with request.state as legacy fallback. That single change closes a quiet pre-existing gap: service-layer commit emissions used to ship organization_id=null on the Redis envelope because the service had no Request. Today AuthScope propagates through nested async work, so every emitted EventMessage — read or commit, router or service — carries the full (organization_id, workspace_id, project_id, user_id).

What changed under the hood

Event type surface. EventType (api/oss/src/core/events/types.py) grows by 29 members across four groups: traces.{fetched,queried}, testcases.{fetched,queried}, and {applications,queries,testsets,evaluators,environments}.revisions.{retrieved,fetched,queried,logged,committed}. WebhookEventType (api/oss/src/core/webhooks/types.py) mirrors the new event types as a strict subset of EventType, derived by .value re-export so the two enums stay in lockstep. The pre-existing environments.revisions.committed precedent is preserved end-to-end (state / diff / references / message attributes intact). Workflow revision events are intentionally deferred until workflows are confirmed as a durable exposed entity; SpansRouter and the legacy TracingRouter are intentionally not instrumented.

Helper layer (core/events/utils.py). Five keyword-only publishers: publish_trace_fetched, publish_trace_queried, publish_testcase_fetched, publish_testcase_queried, and a generic publish_revision_event(domain, action, ...) that fans out to the 25 revision events. Each helper resolves scope (request_scope() → AuthScope first, then request.state), short-circuits on count == 0 / missing revision / empty list, builds the Event envelope with a generated request_id / event_id and RequestType.UNKNOWN, and calls _safe_publish which runs L1 then publishes. Attribute builders are exposed separately (build_trace_fetched_attributes, build_revision_event_attributes, etc.) so call sites can produce attributes without publishing — used by the existing unit tests. List-shaped events cap references / trace_ids / testcase_ids at MAX_REFERENCES = 1000 while count stays uncapped (so consumers can detect truncation via count > len(references)). Commit events deliberately omit count and may carry an optional message. The single extra= kwarg preserves environments.revisions.committed's historical state / diff attributes through the shared helper.

Router-layer read emission. Eight router classes emit reads at the handler boundary after the response is materialized: TracesRouter.{fetch_trace,fetch_traces,query_traces}, TestcasesRouter.{fetch_testcase,fetch_testcases,query_testcases}, and the retrieve / fetch / query / log handlers on each of ApplicationsRouter / QueriesRouter / TestsetsRouter / EvaluatorsRouter / EnvironmentsRouter. Cached paths (the retrieve_*_revision flows in queries / testsets, which use get_cache / set_cache) emit on cache hit too — the helper call sits after the cache check so the count of "the user asked for this revision" is unaffected by whether the response was served from cache. Trace endpoints that internally fetch spans to materialize traces emit only the trace event (no span event). Query-revision endpoints emit query-revision events, not trace events, even when they internally drive a trace fetch.

Service-layer commit emission. Five commit_*_revision(...) methods publish exactly one commit event each: ApplicationsService, QueriesService, TestsetsService, EvaluatorsService, EnvironmentsService. The delta-commit flow in commit_environment_revision and commit_testset_revision early-returns into a private _commit_*_revision_delta(...) helper that resolves the delta into full data and then re-enters commit_*_revision once, so the emission line at the bottom of each method fires exactly once even on delta commits. The existing environments.revisions.committed producer was normalized to use publish_revision_event(extra={"state": ..., "diff": ...}, message=...) instead of constructing the Event envelope inline, preserving every historical attribute. Higher-level compatibility paths (simple-service create / edit, deploy paths from the workflows and evaluators routers, defaults seeding, fork) keep flowing through the service-layer commit method without code change and automatically emit once per call.

L1 quota (publisher side, silent drop). _check_l1_events_quota inside _safe_publish runs check_entitlements(key=Counter.EVENTS_INGESTED, delta=1, cache=True, scope=scope_from(organization_id=...)) when is_ee() and the scope carries an organization_id. Over-quota orgs log [EVENTS] L1 quota exceeded, dropping <event_type> and return without publishing — no HTTP error, no 429. The user's read or commit response is unaffected by event metering. On any other exception the helper fails open (return True) and the publish proceeds; the authoritative check in the worker is the source of truth. OSS skips L1 entirely (the EE imports are deferred under if is_ee():).

L2 quota (worker side, authoritative). EventsWorker.process_batch regroups the per-project batches by org, sums the per-org delta, and runs one check_entitlements(key=Counter.EVENTS_INGESTED, delta=delta, scope=scope_from(organization_id=org_id)) per org (cache=False, the default — atomic DB upsert). Over-quota orgs drop their batch but messages are still ACKed so the consumer-group PEL doesn't back up. On adapter error (Redis / DB glitch) the worker drops the org's batch conservatively, mirroring the tracing worker's stance. The pre-existing per-project Flag.ACCESS check was removed: that flag is False on Hobby and Pro by design (it gates org-flag mutation in organization_router, not events), so the old check would have silently dropped events for every Hobby/Pro customer.

Flag.AUDIT at the query side. POST /events/query (api/oss/src/apis/fastapi/events/router.py) now runs check, _, _ = await check_entitlements(key=Flag.AUDIT) after the existing Permission.VIEW_SPANS access check; on not check the handler returns NOT_ENTITLED_RESPONSE(Tracker.FLAGS). Default plan map: Hobby=False, Pro=False, Business=True, Agenta=True, Self-hosted Enterprise=True. The flag is included in CONSTRAINTS[BLOCKED] so an org can't promote itself. Webhook subscription delivery is not gated by this flag; the audit log and the webhook delivery surface are independent consumers of the same streams:events topic.

AuthScope-first scope resolution. _Scope widened to carry workspace_id in addition to organization_id / project_id / user_id. request_scope() now reads the ambient AuthScope first (via get_auth_scope() in oss/src/utils/context.py) and falls back to request.state only when no auth context is published (admin endpoints, public endpoints, unit tests that hand-craft SimpleNamespace state). publish_revision_event follows the same order: explicit kwargs → AuthScope → request.state. Service-layer commits no longer need to thread organization_id through service signatures — they pick it up from the AuthScope set by the auth middleware on the originating HTTP request, propagated through nested async work via the standard ContextVar mechanism.

Plan map. Counter.EVENTS_INGESTED defaults across the five default plans: Hobby Quota(retention=Retention.MONTHLY, period=Period.MONTHLY), Pro QUARTERLY, Business YEARLY, Agenta and Self-hosted Enterprise period=Period.MONTHLY with no retention (unlimited). Flag.AUDIT defaults: Hobby False, Pro False, Business True, Agenta True, Self-hosted Enterprise True. Both new gates are wired into DEFAULT_ENTITLEMENTS so operator overrides via AGENTA_ACCESS_PLANS / AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY flow through unchanged.

Tests

	acceptance	integration	unit
web	+0 / ~0 / -0	+0 / ~0 / -0	+0 / ~0 / -0
services	+0 / ~0 / -0	+0 / ~0 / -0	+0 / ~0 / -0
api	+0 / ~0 / -0	+0 / ~0 / -0	+48 / ~1 / -0
sdk	+0 / ~0 / -0	+0 / ~0 / -0	+0 / ~0 / -0

Unit additions (api/oss/tests/pytest/unit/events/):

• test_events_utils.py (+~30 new cases on top of the 30 originally shipped): EventType / WebhookEventType parity, attribute builders + capping, revision attribute construction across all five domains, request_scope semantics, AuthScope-first precedence, L1 quota allow / drop / fail-open / skip-on-OSS / skip-when-org-unknown.
• test_service_commit_emission.py (existing): one commit-event-per-call across all five services, environment-specific state / diff preservation, DAO-returns-None suppression, and delta-commit exactly-once coverage for environments and testsets.
• test_events_worker_l2.py (new): per-org delta aggregation across projects, allow / deny / skip-on-OSS / check-failure-drop semantics.
• test_events_router_audit.py (new): direct POST /events/query router coverage for the Flag.AUDIT allow and deny branches.

One edit in api/oss/tests/pytest/unit/test_environments_service.py repoints a patch("oss.src.core.environments.service.publish_event") to patch("oss.src.core.events.utils.publish_event") to match the normalized emission seam. Full OSS unit suite green: 575 passing.

Migrations

No migration in this PR. The events table existed already, the meters_type Postgres enum was extended with events_ingested on the meters reshape (9d3e8f0a1b2c_reshape_meters_table from the feat/clean-up-meters branch, merged into this branch but not part of this PR's design scope), and Flag.AUDIT is a code-level Pydantic enum — no DB representation, no schema change. Plan-map changes ship as code defaults; operators consume them via the existing AGENTA_ACCESS_PLANS / AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY env-driven config layer.

Env vars

No env vars added, edited, or removed. Operators have nothing to change in their .env for this PR. The new Flag.AUDIT and the Counter.EVENTS_INGESTED quota are wired into DEFAULT_ENTITLEMENTS and flow through the existing env-overlay surface; operators who want to override the per-plan defaults use AGENTA_ACCESS_PLANS / AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY as they would for any other entitlement.

Tests and docs

docs/designs/extend-events-beyond-deployments/ carries the design story: research.md and gap.md as historical baselines, proposal.md for the as-shipped read/commit split and AuthScope-first scope resolution, events.md as the per-event reference (29 events with example payloads, capping rules, and delta-commit semantics), tasks.md as the execution checklist, and this summary.md. User-facing docs: docs/docs/prompt-engineering/integrating-prompts/04-webhooks.mdx lists every subscribable event type and now distinguishes read/log payloads (count) from commit payloads (no count, optional message); the generated Fern TS client (web/packages/agenta-api-client/src/generated/api/types/WebhookEventType.ts) and docs/docs/reference/openapi.json include all 29 new types. Internal API guidance (AGENTS.md) is unchanged — the helper layer follows the existing domain-folder + DTO-mapping conventions.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	May 19, 2026 7:26pm

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c46609c7-f91a-4928-997a-7d4ca6d33d1c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/extend-events-beyond-deployments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR extends Agenta's webhook event system from a single deployment-related event to a comprehensive set of events covering traces, testcases, and revision lifecycles (retrieve/fetch/query/log/committed) for applications, queries, testsets, evaluators, and environments. It introduces shared helpers in core/events/utils.py with a deliberate read-at-router / write-at-service split, instruments the relevant routers and service-layer commit methods, and regenerates the OpenAPI/Fern client artifacts plus docs.

Changes:

• Adds 28 new EventType/WebhookEventType values and shared publish_* helpers (with 1000-item caps and zero-count suppression).
• Instruments router read-paths (TracesRouter, TestcasesRouter, and the four revision routers) and refactors EnvironmentsService plus adds emission to four new commit_*_revision services.
• Regenerates docs (OpenAPI status/request schemas, MDX pages migrating to Docusaurus :::caution[deprecated] and bare <ParamsDetails>) and Python client WebhookEventType literal, plus design docs.

Reviewed changes

Copilot reviewed 201 out of 203 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
api/oss/src/core/events/types.py, core/webhooks/types.py	Adds the 28 new event-type enum members.
api/oss/src/core/events/utils.py (context)	New shared publish_* helpers documenting the router/service split.
api/oss/src/core/{applications,queries,testsets,evaluators,environments}/service.py	Emits *.revisions.committed from each commit_*_revision; environments refactored to use the shared helper.
api/oss/src/apis/fastapi/{traces,testcases,applications,queries,testsets,evaluators,environments}/router.py	Emits read events (fetched/queried/retrieved/log) after responses are materialized.
clients/python/agenta_client/types/webhook_event_type.py	Updates the Literal[...] union to include the new event names.
docs/docs/reference/api/*.{api.mdx,StatusCodes.json,RequestSchema.json}	Regenerated OpenAPI docs: new event enums, :::caution[deprecated] syntax, <ParamsDetails> cleanup.
docs/docs/prompt-engineering/integrating-prompts/04-webhooks.mdx	Documents the new event taxonomy.
docs/designs/extend-events-beyond-deployments/{tasks,gap,findings}.md	Design notes capturing the implementation decisions and deferred acceptance tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Railway Preview Environment

Status	Destroyed (PR converted to draft)

Updated at 2026-05-19T06:49:31.717Z

[Copilot]

Pull request overview

Copilot reviewed 45 out of 47 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 59 out of 61 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 59 out of 61 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 59 out of 61 changed files in this pull request and generated no new comments.