What this unlocks

This PR turns the previously code-locked plan/role/catalog/pricing surface into an env-driven configuration layer. Operators may set AGENTA_ACCESS_PLANS, AGENTA_ACCESS_ROLES, AGENTA_ACCESS_ROLES_OVERLAY, AGENTA_ACCESS_DEFAULT_PLAN, AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY, AGENTA_BILLING_CATALOG, and AGENTA_BILLING_PRICING at deployment time, restart the API and worker processes, and the new shape is the effective shape — no code change, no rebuild. When any of the JSON env vars are absent the system falls back to code defaults, so existing deployments need no operator action.

The immediate capabilities this lights up are: define a custom plan with a custom entitlement profile (new counter limits, new gauges, new throttle buckets, new flags), tweak a single quota on an existing plan without restating the rest (AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY merges field-by-field into whichever plan AGENTA_ACCESS_DEFAULT_PLAN points to), add a custom role at the project (and workspace, by symmetry) scope without restating the platform-managed minima or the default workspace roles (AGENTA_ACCESS_ROLES_OVERLAY), mark the free and trial plan slugs inline in pricing ({free: true} / {trial: N} markers on the pricing entries themselves — no cross-env-var consistency burden), and separate display catalog from purchase pricing so the pricing modal and the Stripe line-item / per-meter-price configuration evolve independently. The closed Plan enum is gone; subscriptions.plan is a plain str, and the auth-side role serialization widened to str too — env-defined custom roles flow cleanly through WorkspacePermission.role_name and InviteRequest.roles without enum-coercion failures.

A second hard cut shipped alongside: events retention split out of billing. The old POST /admin/billing/usage/flush (spans-only) is gone; spans now flush at POST /admin/spans/flush (cron crons/spans.sh at 0,30 * * * *, lock namespace spans:flush) and events flush at POST /admin/events/flush (cron crons/events.sh at 7,37 * * * *, lock namespace events:flush). They share nothing — separate DAOs, services, routers, cron files, Redis locks — so the two flushes can run concurrently and a failure in one never blocks the other. Counter.EVENTS_INGESTED was added to the entitlement system as a retention-only counter (no write-path adjusts a meter row for it, no entry in the meters_type Postgres enum, deliberately not in REPORTS); the events flush job walks the effective plan map and respects each plan's Counter.EVENTS_INGESTED.retention. Per-plan defaults align with each plan's Counter.TRACES_INGESTED retention: Quota(period=Period.MONTHLY, retention=Retention.MONTHLY) on Hobby, QUARTERLY on Pro, YEARLY on Business, and retention=None (unlimited) on Agenta and Self-hosted Enterprise.

What changed under the hood

env vars layout. api/oss/src/utils/env.py gains two sub-models on EnvironSettings: AccessControls (plans, roles, roles_overlay, default_plan, default_plan_overlay) and BillingSettings (catalog, pricing). The JSON-bearing fields are decoded into typed Python objects at startup via small _load_json_env_* helpers, so downstream modules consume already-parsed dicts / lists and never re-parse strings. AGENTA_ACCESS_DEFAULT_PLAN is the canonical default-plan name; legacy AGENTA_DEFAULT_PLAN is honored as a read-through fallback so existing deployments are not forced to rename their env var. The trial-flow config previously carried by AGENTA_BILLING_TRIAL_PLAN / AGENTA_BILLING_TRIAL_DAYS is now inline on the pricing entry as a {trial: N} marker (positive integer days) — there can be at most one trial entry across the whole pricing dict, and BillingSettings no longer carries trial_plan / trial_days fields.

controls.py — the access-controls builder. A new module api/ee/src/core/entitlements/controls.py owns plans + roles. _build_controls() runs once at import time: it parses AGENTA_ACCESS_PLANS against _PlanOverride (Pydantic, extra="forbid") or falls back to DEFAULT_ENTITLEMENTS keyed by the DefaultPlan enum, applies AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY to the resolved default plan via _merge_quota / _merge_throttle, parses AGENTA_ACCESS_ROLES and the role catalog falls back to _default_roles() (organization gets minima only, workspace and project get minima + default extras like admin / developer / editor / annotator), applies AGENTA_ACCESS_ROLES_OVERLAY to both workspace and project scopes, and hashes the resulting controls into a 12-character SHA stamp. The public surface — get_plans(), get_plan(slug), get_plan_entitlements(slug), get_plan_description(slug), get_roles(scope), get_role(scope, slug), get_role_permissions(scope, slug), get_role_description(scope, slug), get_controls_hash() — returns the frozen effective state and is what every runtime caller uses. At startup the module logs [access-controls] plans=defaults|env roles=defaults|env plan_overlay=none|env→<slug> roles_overlay=none|env hash=<12char> so multi-worker deployments can grep across logs to verify every process loaded the same config.

settings.py — the billing-settings builder. api/ee/src/core/subscriptions/settings.py parses AGENTA_BILLING_CATALOG (list of _CatalogEntry Pydantic models with extra="allow" so operators can add display-only fields the frontend renders; type ∈ {"standard", "custom"} is enforced) and AGENTA_BILLING_PRICING (object keyed by plan slug — flat shape {slug: {free?, trial?, <stripe_slot>: {price, quantity?}}} that mirrors the original STRIPE_PRICING plus the two reserved markers). Reserved keys are free (bool) and trial (positive int days); every other key on a pricing entry is a Stripe-side slot name owned by the operator (the same names they used on the Stripe dashboard — typically "traces" and "users") and is not validated against an internal enum. Cross-cutting validation runs in _build_settings(): every catalog plan slug must exist in the effective plan set, every pricing slug must exist in the effective plan set, at most one entry may carry "free": true, at most one entry may carry "trial": N, the free-plan fallback must be resolvable, and AGENTA_ACCESS_DEFAULT_PLAN (when set) must be in the effective plan set. Accessors get_catalog(), get_catalog_plan(slug), get_pricing(), get_pricing_plan(slug), get_stripe_line_items(slug), get_stripe_meter_price(plan, meter), get_free_plan(), get_trial_plan(), get_trial_days(), trial_enabled() are how the billing router, the meter reporting job, the subscription service, and the signup flow read the effective settings. [billing-settings] catalog=defaults|env pricing=defaults|env free_plan=<slug>|None trial=<slug>/<n>d|disabled is logged at startup.

Stripe meter name mapping. Internal Counter / Gauge slugs (code identifiers) and Stripe-side meter event names (external configuration the operator owns on the Stripe dashboard) are independent surfaces. api/ee/src/core/entitlements/types.py carries a single source of truth at module scope: STRIPE_METER_NAMES: dict[str, str] = {Counter.TRACES_INGESTED.value: "traces", Gauge.USERS.value: "users"}. The runtime in meters/service.py uses STRIPE_METER_NAMES.get(meter.key.value) for both the stripe.billing.MeterEvent.create(event_name=...) argument and the per-meter price-id lookup; meters not in the map fall through the existing skipped-meter log path with a clear message. The pricing-config slot names in AGENTA_BILLING_PRICING agree with the Stripe-side names (operator-owned), so an internal rename of Counter.TRACES_INGESTED does not require a coordinated Stripe-dashboard change or an env-var rewrite on every deployment. Adding a new Stripe-reportable counter or gauge requires exactly two edits: add the Counter / Gauge member to REPORTS, and add the row to STRIPE_METER_NAMES.

Default-plan overlay. AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY lets self-hosted operators tweak individual entitlement values on the default plan without restating the rest. The shape mirrors _PlanOverride (description, flags, counters, gauges, throttles) with one divergence: throttles is a map keyed by category slug instead of a list, so per-category patches don't require restating the whole throttle list. Merge semantics — description replaces, flags per-key replace, counters/gauges field-merge inside each quota (overlay-set fields replace, omitted fields keep the base; pass null to clear), and throttles[category] looks up the existing single-category throttle on the base plan and field-merges its bucket. Operators who need multi-category or endpoint-keyed throttle changes use AGENTA_ACCESS_PLANS instead. Resolution happens after the base plan map is built but before the public dicts are frozen, so all downstream accessors see the merged state.

Roles overlay. AGENTA_ACCESS_ROLES_OVERLAY is the symmetric overlay for the role catalog. Today only the project top-level key is accepted; the patch is applied to both the workspace and project scopes because they share the same default role set in the code defaults. If workspace or organization appears as a key, startup fails — silent ignore would mislead operators. Per role slug: if the slug exists on the scope, the patch is a per-field replace (permissions swaps the array, description swaps the string, fields not set are preserved); if the slug does not exist, the entry is appended as a new role and both permissions and description must be supplied. owner and viewer minima are reserved and cannot be patched — they are platform-managed and synthesized fresh after every override.

Catalog vs. pricing. Splitting them is the central design choice: AGENTA_BILLING_CATALOG is the user-facing pricing-modal display (title, description, type, features, optional display price and retention for marketing copy), and AGENTA_BILLING_PRICING is the Stripe wiring (line items for checkout / subscription create, per-meter price IDs for usage reporting, plus the free / trial markers). Catalog entries without a plan field are exempt from the effective-plan-set cross-check (contact-sales tiers like Enterprise have no slug). Stripe line items are optional on a per-plan basis — custom and self-hosted plans aren't directly purchasable, and paid-checkout flows fail with a clear "Plan 'X' is not available for purchase (no Stripe line items configured)" instead of silently mis-charging. Only counters in the REPORTS allowlist (today traces_ingested and users) actually get pushed to Stripe; declaring price IDs for other slot names is accepted by the parser but those meters are not reported.

Onboarding plan resolution. get_default_plan() reads env.access_controls.default_plan first (with the legacy AGENTA_DEFAULT_PLAN as fallback), and falls back to DefaultPlan.CLOUD_V0_HOBBY when Stripe is enabled and DefaultPlan.SELF_HOSTED_ENTERPRISE when Stripe is disabled. get_free_plan() returns whichever pricing entry is marked "free": true, falling back to DefaultPlan.CLOUD_V0_HOBBY when no env-driven free plan is defined (and only if that slug is in the effective plan set — operators who restrict AGENTA_ACCESS_PLANS to a slug set without cloud_v0_hobby must mark one pricing entry as "free": true or startup fails). trial_enabled() is true when some pricing entry carries a "trial": N marker (the single trial entry's slug + day count drive get_trial_plan() and get_trial_days()). Behavior: Stripe-enabled + trial-configured → reverse-trial flow on the trial plan, then auto-downgrade to free; Stripe-enabled + no trial → direct onboarding on the free plan; Stripe-disabled → direct onboarding on get_default_plan().

Runtime refactor. Direct imports of CATALOG, ENTITLEMENTS, FREE_PLAN, REVERSE_TRIAL_PLAN, REVERSE_TRIAL_DAYS are gone from runtime code. The closed Plan enum is gone; SubscriptionDTO.plan is Optional[str], response models use str, role-validation uses controls.get_role(scope, slug) instead of WorkspaceRole(value). Constants in entitlements/types.py were renamed to DEFAULT_CATALOG / DEFAULT_ENTITLEMENTS to signal their fallback role, and DefaultPlan / DefaultRole enums survive as code-default seeds used only for the fallback and for type-safe conditional checks against well-known slugs. The throttle middleware (api/ee/src/services/throttling_service.py) iterates get_plans() and falls back to the free plan's throttle bucket when an organization is on an unknown plan, so a misconfigured / orphaned subscription still gets rate-limited instead of bypassing throttling entirely. The tracing-flush job and the events-flush job both walk get_plans() and respect each plan's Counter.TRACES_INGESTED.retention / Counter.EVENTS_INGESTED.retention per the new dynamic plan map. Permission.default_permissions(role) callers were replaced with get_role_permissions(scope, role) and WorkspaceRole.get_description(role) callers with get_role_description(scope, role) so the resolution path goes through the effective catalog, not the closed enum.

Quota hardening. Quota, Probe, Bucket, and Throttle Pydantic models all declare model_config = ConfigDict(extra="forbid") so operator typos in AGENTA_ACCESS_PLANS or AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY JSON (the most common one being a leftover "monthly": true from the pre-reshape config) fail startup with a clear Pydantic error pointing at the offending field instead of silently dropping. _merge_quota round-trips through model_dump() + dict update + model_validate(), which preserves enum types (Pydantic v2 coerces JSON strings/ints back to their Period / Retention / Scope enum members) and benefits from the strict config on each merge.

Meters service value-based identity. api/ee/src/core/meters/service.py was switched from name-based identity (meter.key.name in Gauge.__members__.keys()) to value-based (meter.key.value in _GAUGE_SLUGS) where the slug sets are computed once at module scope. The Meters enum (meter table column) and the Counter / Gauge enums (entitlements) share lowercase string values but are independent types — using values rather than names keeps the dispatch correct if either side is ever renamed without touching the other. The same module-level frozensets _GAUGE_SLUGS and _COUNTER_SLUGS are reused by both the gauge-update and counter-event branches of the Stripe report loop.

Catalog → Stripe price wiring. A small helper script docs/designs/dynamic-access-and-billing/migrate_stripe_pricing.py annotates an existing flat pricing dict with the free and trial markers (annotate(pricing, free_slug=..., trial=(slug, days))) so operators who already have the original STRIPE_PRICING shape can derive the new AGENTA_BILLING_PRICING form with a single pass. Slot names ("traces", "users") are operator-owned and pass through unchanged. The legacy env vars are no longer read by the runtime — only the new AGENTA_BILLING_PRICING is consulted.

Auth-context wiring. AGENTA_ACCESS_* overrides do not touch the auth path directly, but the access-control resolution functions get_role(scope, slug) and get_role_permissions(scope, slug) are what api/ee/src/utils/permissions.py consults at every permission-check call site. WorkspaceRole is preserved in the codebase as a code-default seed used only by _default_roles(); runtime resolution goes through the effective catalog. Organization-scope viewer has no permissions by design (orgs are membership markers); workspace and project viewer get the code-default read-only permission set so existing project members keep resolving to their historical permission sets. Env overrides may add roles or customize non-minima permissions but cannot remove or rebind the owner/viewer minima — they are re-synthesized after every override.

Tests

Migrations

One Alembic revision on this branch: a1b2c3d4e5f7_unify_org_member_role_to_viewer.py. It unifies the organization_members.role server-side default and any existing "member" rows to "viewer", completing the org-role rename in OrganizationRole so every scope (organization / workspace / project) shares the same least-permission role slug. Downgrade restores the legacy "member" default and re-maps rows back. Chained after the meters reshape (9d3e8f0a1b2c) so the EE core Alembic tree stays single-headed after the feat/clean-up-meters merge.

Two existing migrations were edited:

• 7990f1e12f47_create_free_plans.py: comment annotated with the operator constraint on cloud_v0_hobby. The startup-time guard in subscriptions/settings.py makes a violation fail fast at boot rather than silently corrupt subscription rows.
• a9f3e8b7c5d1_clean_up_organizations.py: same comment annotation.

The companion meters reshape (9d3e8f0a1b2c_reshape_meters_table.py) shipped on the separate feat/clean-up-meters branch and is documented under docs/designs/extend-meters/; it is merged into this branch but is not part of this PR's design scope.

Env vars

Added (all optional — code defaults apply when unset):

• AGENTA_ACCESS_PLANS — JSON object keyed by plan slug; values are _PlanOverride-shaped (description, flags, counters, gauges, throttles).
• AGENTA_ACCESS_ROLES — JSON object keyed by scope (organization/workspace/project); values are non-empty arrays of role entries.
• AGENTA_ACCESS_ROLES_OVERLAY — JSON object with a single project key; values are role-slug → {permissions?, description?} patches applied to both workspace and project scopes.
• AGENTA_ACCESS_DEFAULT_PLAN — plan slug for signup; legacy AGENTA_DEFAULT_PLAN honored as fallback.
• AGENTA_ACCESS_DEFAULT_PLAN_OVERLAY — JSON object mirroring a plan entry's shape; field-merged into the resolved default plan at startup.
• AGENTA_BILLING_CATALOG — JSON array of catalog entries served by /billing/plans.
• AGENTA_BILLING_PRICING — JSON object keyed by plan slug; flat shape with operator-owned Stripe slot names (traces, users, etc.) plus reserved free: bool and trial: int markers.

Removed: legacy STRIPE_PRICING / AGENTA_PRICING are no longer read; AGENTA_BILLING_TRIAL_PLAN / AGENTA_BILLING_TRIAL_DAYS are collapsed into the per-entry {trial: N} marker on AGENTA_BILLING_PRICING. migrate_stripe_pricing.py in this design folder annotates an existing flat pricing dict with the free / trial markers.

Tests and docs

Unit tests live in api/ee/tests/pytest/unit/: test_access_controls.py (parser-level tests for plans, roles, and overlay merge — including reserved-slug rejection, duplicate-slug rejection, unknown-permission rejection, throttle-category mismatch rejection — 61 tests), test_controls_env_override.py (subprocess-driven end-to-end env override tests covering every JSON env var across the no-override / override / overlay states), test_billing_settings.py (catalog + pricing parsers under the flat shape, _resolve_trial per-entry trial detection, free-plan validation, default-plan validation), test_billing_router.py (billing handler behavior under env overrides), test_events_retention.py (events flush service: plan iteration, project pagination, per-plan failure isolation), test_admin_retention_routers.py (spans + events admin endpoints — lock namespaces, busy-lock skip, handler shape). EE unit suite green across the three primary access/billing test files: 61 + 83 = 144 tests.

docs/designs/dynamic-access-and-billing/ carries the design story: research.md and gap.md as historical baselines, proposal.md for the as-shipped state, tasks.md as the execution checklist (initial + post-initial sections covering the default-plan relocation, events counter + retention split, and roles overlay), migrate_stripe_pricing.py as the free / trial annotation helper, and this summary.md. Operator-facing docs cover access controls only: docs/docs/self-host/04-dynamic-access-controls.mdx documents every AGENTA_ACCESS_* env var with field tables, examples, and validation rules; docs/docs/self-host/02-configuration.mdx carries the configuration-table row pointing at those variables. Billing / Stripe wiring is internal-only and not documented in the operator-facing MDX. The hosting/docker-compose/ee/env.ee.dev.example and env.ee.gh.example files were extended with the new AGENTA_ACCESS_* variables; no billing / Stripe env vars appear in either file.