Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

44 changes: 37 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Link.Assistant.Router is a transparent proxy that sits between API clients (such
- **Optional Gonka upstream** — `UPSTREAM_PROVIDER=gonka` forwards OpenAI-compatible routes to Gonka instead of translating them to Anthropic
- **Optional Crater ForgeFed upstream** — `UPSTREAM_PROVIDER=crater` turns OpenAI chat requests into ForgeFed `Offer{Ticket}` tasks and waits for resolved task results
- **Optional LiteLLM/OpenAI-compatible upstream** — `UPSTREAM_PROVIDER=openai-compatible` routes OpenAI SDK traffic to a stored provider such as LiteLLM
- **Multi-account routing** — pool any number of Claude MAX accounts; round-robin / priority / least-used; automatic cooldowns on 429
- **Multi-account routing** — pool any number of Claude, Codex, Gemini, or Qwen subscriptions; session affinity, strict token pins, round-robin / fill-first / least-used selection, request caps, and `Retry-After`-aware cooldowns
- **Issues custom `la_sk_...` JWT tokens** with expiration and revocation for multi-tenant access
- **Persistent token store** — text (Lino) **and** binary backends, both on by default; tokens survive restarts
- **Live observability** — Prometheus `/metrics`, JSON `/v1/usage`, per-account health at `/v1/accounts`
Expand Down Expand Up @@ -83,6 +83,15 @@ Gemini uses the Code Assist envelope with synthesized SSE for streaming; Qwen is
OpenAI-compatible). Run `router doctor` to verify each credential file is
present and its token valid.

To pool subscriptions, set `ADDITIONAL_ACCOUNT_DIRS` to vendor-specific
credential homes. The active `UPSTREAM_PROVIDER` determines how every directory
is parsed. New sessions use `ACCOUNT_ROUTING_STRATEGY`; a session remains on its
chosen account for `SESSION_AFFINITY_TTL_SECS`, and an `la_sk_...` token issued
with an `account` claim is a strict pin. Automatic selection skips accounts
whose configured `ACCOUNT_REQUEST_LIMITS` cap is spent or whose upstream
returned HTTP 429. Pinned and session-bound requests fail instead of silently
changing identity.

## Quick Start

### Prerequisites
Expand Down Expand Up @@ -281,6 +290,8 @@ Claude Code will work exactly as normal, with all requests transparently proxied
|---|---|---|
| `/v1/messages` | POST | Anthropic Messages — preserves SSE streaming |
| `/v1/messages/count_tokens` | POST | Token-count helper |
| `/api/anthropic/v1/messages` | POST | Namespaced Anthropic Messages alias |
| `/api/anthropic/v1/messages/count_tokens` | POST | Namespaced token-count alias |
| `/invoke` | POST | Bedrock-format invoke |
| `/invoke-with-response-stream` | POST | Bedrock streaming invoke |
| `/api/latest/anthropic/*` | ANY | Legacy prefix; stripped and forwarded |
Expand All @@ -293,6 +304,19 @@ Claude Code will work exactly as normal, with all requests transparently proxied
| `/v1/chat/completions` | POST | Chat Completions, translated to Anthropic Messages, forwarded to the selected OpenAI-compatible provider, or delivered as a Crater ForgeFed task |
| `/v1/responses` | POST | Responses API, translated to Anthropic Messages or forwarded to the selected OpenAI-compatible provider |
| `/v1/models` | GET | OpenAI-shaped model list |
| `/api/openai/v1/*` | GET/POST | Namespaced aliases for models, Chat Completions, and Responses |
| `/api/codex/v1/*` | GET/POST | Codex namespace; Responses is the subscription's native protocol |
| `/api/qwen/v1/*` | GET/POST | Qwen namespace; forwards its native OpenAI-compatible protocol |
| `/api/gemini/v1beta/models` | GET | Native Gemini model list |
| `/api/gemini/v1beta/models/{model}` | GET | Native Gemini model metadata |
| `/api/gemini/v1beta/models/{model}:generateContent` | POST | Native Gemini generation |
| `/api/gemini/v1beta/models/{model}:streamGenerateContent` | POST | Native Gemini SSE response |
| `/api/vertex/v1/projects/.../models/{model}:generateContent` | POST | Native Vertex-style generation through Gemini Code Assist |

Provider-specific namespaces use the matching active subscription configured
by `UPSTREAM_PROVIDER`; for example, native Gemini and Vertex generation
requires `UPSTREAM_PROVIDER=gemini`. They are additive client-facing protocol
aliases, not cross-provider fallback rules.

`gpt-4o`, `gpt-4o-mini`, `gpt-4`, and the `o*` reasoning families auto-map to the Claude Sonnet / Haiku / Opus tiers respectively. Native `claude-*` IDs pass through unchanged.

Expand Down Expand Up @@ -337,7 +361,7 @@ method-specific verifier is configured.
|---|---|---|
| `/metrics` | GET | Prometheus text-exposition counters |
| `/v1/usage` | GET | JSON snapshot of all counters |
| `/v1/accounts` | GET | Multi-account health: cooldowns, last error, used-count |
| `/v1/accounts` | GET | Multi-account health: cooldowns, last error, used count, configured limit, and remaining requests |

### POST /api/tokens

Expand Down Expand Up @@ -422,7 +446,7 @@ Every flag listed in `--help` has an env-var alias and can be configured from
| `--port` / `ROUTER_PORT` | `8080` | No | Port to listen on |
| `--host` / `ROUTER_HOST` | `0.0.0.0` | No | Host/IP to bind to |
| `--claude-code-home` / `CLAUDE_CODE_HOME` | `~/.claude` | No | Primary Claude Code credentials directory |
| `--upstream-provider` / `UPSTREAM_PROVIDER` | `anthropic` | No | Upstream provider: `anthropic`, `gonka`, `crater`, or `openai-compatible` |
| `--upstream-provider` / `UPSTREAM_PROVIDER` | `anthropic` | No | Upstream provider: `anthropic`, `codex`, `gemini`, `qwen`, `gonka`, `crater`, or `openai-compatible` |
| `--upstream-base-url` / `UPSTREAM_BASE_URL` | `https://api.anthropic.com` | No | Upstream Anthropic API URL |
| `--api-format` / `UPSTREAM_API_FORMAT` | (auto) | No | Restrict the proxy to `anthropic` / `bedrock` / `vertex` |
| `--verbose` / `VERBOSE` | `false` | No | Verbose tracing |
Expand Down Expand Up @@ -550,7 +574,11 @@ The HTTP API accepts the same shape at `POST /api/providers`:
| `--storage-policy` / `STORAGE_POLICY` | `both` | Persistent token store: `memory`, `text` (Lino), `binary`, or `both` |
| `--data-dir` / `DATA_DIR` | platform-specific | Where `tokens.lino` / `tokens.bin` live |
| `--claude-cli-bin` / `CLAUDE_CLI_BIN` | `claude` | Local Claude CLI binary used by the `cli` backend |
| `--additional-account-dirs` / `ADDITIONAL_ACCOUNT_DIRS` | (empty) | Comma-separated extra credential dirs for multi-account routing |
| `--additional-account-dirs` / `ADDITIONAL_ACCOUNT_DIRS` | (empty) | Comma-separated extra credential homes for the active subscription provider |
| `--account-routing-strategy` / `ACCOUNT_ROUTING_STRATEGY` | `round-robin` | New-session policy: `round-robin`, `priority`/`fill-first`, or `least-used`/`quota-first` |
| `--account-cooldown-secs` / `ACCOUNT_COOLDOWN_SECS` | `60` | Minimum cooldown after a quota response; a longer upstream `Retry-After` wins |
| `--session-affinity-ttl-secs` / `SESSION_AFFINITY_TTL_SECS` | `3600` | Inactive seconds before a conversation can be assigned again; `0` disables affinity |
| `--account-request-limits` / `ACCOUNT_REQUEST_LIMITS` | (unknown) | Comma-separated request caps, primary first then extras; must match pool size, and `0` means unknown/unlimited |

### Feature toggles

Expand Down Expand Up @@ -777,9 +805,9 @@ curl -s -X POST http://localhost:8080/api/tokens \
cargo test
```

This runs:
- **Unit tests** in every module under `src/` (44 tests covering config, oauth, token, storage, accounts, openai, metrics, cli)
- **Integration tests** in `tests/integration_test.rs` cover API path routing, OpenAI translation, metrics rendering, and CLI parsing
This runs the unit, integration, release-workflow, and documentation tests,
including account affinity/caps, provider-scoped token caching, request-routing
metadata, protocol translation, metrics, and configuration validation.

### Run specific test suites

Expand Down Expand Up @@ -891,9 +919,11 @@ This demonstrates token issuance, validation, and revocation programmatically.
│ ├── crater.rs # Crater ForgeFed task provider
│ ├── oauth.rs # Claude Code OAuth credential reader
│ ├── accounts.rs # Multi-account router (round-robin/priority/least-used + cooldowns)
│ ├── app_state.rs # Shared HTTP handler state
│ ├── storage.rs # Persistent token store (text Lino + binary backends)
│ ├── providers.rs # OpenAI-compatible provider store + encrypted secrets
│ ├── proxy.rs # Transparent API proxy with token swap, OpenAI shim, ops endpoints
│ ├── request_routing.rs # Session/account routing signal extraction
│ ├── openai.rs # OpenAI <-> Anthropic translation helpers
│ ├── metrics.rs # Atomic counters, Prometheus rendering, JSON snapshots
│ └── token.rs # Custom JWT token management (la_sk_...)
Expand Down
20 changes: 20 additions & 0 deletions changelog.d/20260722_070000_issue_42_multi_subscription_routing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
bump: minor
---

### Added

- Provider-neutral multi-subscription pools for Claude, Codex, Gemini, and Qwen
with strict token pins, session affinity, round-robin/fill-first/least-used
selection, configurable per-account request caps, and `Retry-After`-aware
quota cooldowns.
- Formal AI-style namespaced protocol routes for Anthropic, OpenAI, Codex,
Qwen, native Gemini `generateContent`, and Vertex publisher-model requests.
- Issue #42 research and requirement trace under
`docs/case-studies/issue-42`.

### Fixed

- Subscription requests now enforce router-token request budgets, keep
refreshed credentials isolated per account, and preserve original request
metadata when selecting an account before protocol translation.
126 changes: 126 additions & 0 deletions docs/case-studies/issue-42/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Issue 42: Multi-subscription routing

## Scope

Issue [#42](https://github.com/link-assistant/router/issues/42) asks the router
to adopt Claudexor's multi-subscription practices, support both single- and
multi-account deployments for every existing vendor subscription, minimize
spent capacity when selecting new work, preserve sessions, and expose the
provider-native API namespaces used by Formal AI.

The implementation extends the existing `SubscriptionProvider` abstraction;
it does not create a second vendor registry. Claude, Codex, Gemini, and Qwen
therefore share one account-selection policy while retaining their existing
credential readers, refresh flows, request translations, and upstreams.

## Root cause

PR #38 added four subscription readers, but the account pool was still built
from Claude-only `OAuthProvider` instances. Codex, Gemini, and Qwen bypassed the
pool, refreshed tokens shared a cache slot per provider instead of per account,
and routing happened before the proxy copied stable session metadata. The
result was four usable single subscriptions but only one usable multi-account
provider, with no identity-stability contract.

The public HTTP surface also exposed generic Anthropic/OpenAI paths only. A
Gemini client could reach Code Assist only after translating through the OpenAI
shape, even though Formal AI demonstrates that native protocol namespaces are
useful integration boundaries.

## Adopted design

### One provider-neutral pool

`AccountRouter::new_for_provider` builds an ordered pool of
`SubscriptionReader`s for the active provider. A pool is optional: the existing
single-account path remains the default when no extra directories or account
caps are configured.

New work supports three policies:

- `round-robin` distributes requests across available accounts;
- `priority` / `fill-first` keeps using the first available account;
- `least-used` / `quota-first` compares normalized configured consumption
(`used / limit`) and chooses the lowest spent account. Unknown limits remain
eligible fallback capacity instead of being presented as zero usage.

`ACCOUNT_REQUEST_LIMITS` supplies the router's internal per-account limits,
ordered primary then additional accounts. A value of zero explicitly means
unknown/unlimited. Counters are atomic, and the cap check plus increment is a
compare-and-swap operation so concurrent requests cannot oversubscribe a cap.

### Stable and strict identity

The router copies routing signals before request translation:

1. a router-issued token's `account` claim is an explicit strict pin;
2. session headers (`x-claude-code-session-id`, `x-codex-session-id`,
`x-session-id`, `session-id`) win over body metadata;
3. JSON session/conversation metadata and OpenAI `prompt_cache_key` are
accepted as fallback session keys;
4. requests without either signal enter the configured automatic policy.

Session bindings expire after `SESSION_AFFINITY_TTL_SECS` of inactivity. A
pinned or bound account that becomes spent/unavailable fails selection; the
router does not move an in-flight conversation to a different identity.

### Quota failure behavior

Only an upstream HTTP 429 starts account cooldown. `Retry-After` is parsed in
both delta-seconds and HTTP-date forms, and the longer of that value and
`ACCOUNT_COOLDOWN_SECS` wins. Concurrent failures can extend but never shorten
an existing cooldown. Network, parsing, model, and request-shape errors do not
rotate an account.

The account-scoped refresh cache key is `(provider, account name)`. This avoids
returning account A's refreshed credential after account B has been selected.
Per-token request budgets are now enforced on subscription paths as well as the
Anthropic proxy path.

### Native protocol namespaces

The unprefixed compatibility routes remain stable. The following additive
namespaces mirror Formal AI's protocol organization:

| Namespace | Routes | Native role |
| --- | --- | --- |
| `/api/anthropic/v1` | Messages, count tokens | Anthropic subscription protocol |
| `/api/openai/v1` | models, Chat Completions, Responses | Generic OpenAI compatibility |
| `/api/codex/v1` | models, Chat Completions, Responses | Codex's native Responses upstream plus compatibility projection |
| `/api/qwen/v1` | models, Chat Completions, Responses | Qwen/DashScope's native OpenAI-compatible upstream |
| `/api/gemini/v1beta` | models, model metadata, `generateContent`, `streamGenerateContent` | Native Gemini request/response bodies |
| `/api/vertex/v1` | publisher-model generation paths | Native Vertex-style request paths over Gemini Code Assist |

Each provider-specific namespace uses the matching configured
`UPSTREAM_PROVIDER`; namespaces do not introduce cross-provider fallback.

Native Gemini streaming currently returns the complete Code Assist response as
one valid SSE data event. This preserves the native response shape and stream
transport without claiming token-by-token upstream streaming.

## Verification

The regression suite covers:

- same-session account affinity and strict no-fallback behavior;
- strict explicit account claims and unknown pins;
- all three automatic policies and normalized least-used selection;
- atomic configured limits, spent-account exclusion, cooldown extension, and
`Retry-After` parsing;
- Codex pool selection and per-account refresh-cache isolation;
- token account lookup, request metadata extraction, and subscription budget
enforcement;
- Gemini/Vertex native path parsing and response-envelope handling;
- configuration validation, including exact cap-to-pool cardinality.

Before implementation, compiling the new regression tests produced 26 missing
API/type errors. After implementation, the complete `cargo test --all-features`
suite passes, and a real `cargo run -- serve` startup confirms that the Axum
route table has no conflicting paths.

## Files in this case study

- [requirements.md](requirements.md) maps every issue requirement to code and
verification.
- [online-research.md](online-research.md) records repositories, immutable
snapshots, inspected files, findings, and design decisions.
Loading
Loading