diff --git a/skills/openrouter-analytics-query/SKILL.md b/skills/openrouter-analytics-query/SKILL.md index f6b7446..9264063 100644 --- a/skills/openrouter-analytics-query/SKILL.md +++ b/skills/openrouter-analytics-query/SKILL.md @@ -82,15 +82,16 @@ cd /scripts && npx tsx query-analytics.ts --met ### Filter Object Shape ```json -{ "field": "", "operator": "", "value": "" } +{ "field": "", "operator": "", "value": "", "include_unset": true } ``` - Scalar operators (`eq`, `neq`, `gt`, `gte`, `lt`, `lte`): `value` is a string or number - Array operators (`in`, `not_in`): `value` is an array of strings or numbers +- `include_unset` is optional and applies only to `in` and `not_in` filters on dimensions with an unset bucket: `api_key_id` (`-1`), `app` (`-1`), and `user` (empty string). With `in`, unset rows are included with the listed values. With `not_in`, unset rows are excluded along with the listed values. An empty `value` array with `in` matches only unset rows. Other dimensions return the validation error `Dimension "" has no unset bucket`. - Several dimensions are **label-resolved** in query results (returned as human-readable names), but filters must use the underlying ID: - `api_key_id` — numeric ID (from generation metadata) or 64-char SHA-256 hash (from `GET /api/v1/keys`). Hashes are auto-resolved to numeric IDs before querying. - `user` — Clerk user ID (e.g. `user_abc123`), not the display name/email shown in results. - - `workspace` — workspace UUID, not the workspace name shown in results. + - `workspace` — workspace UUID, not the workspace name shown in results. Filtering or grouping by the account default workspace also covers activity recorded before workspace resolution existed, which is folded from the legacy all-zero UUID. - `app` — numeric app ID, not the app title shown in results. - `model` — permaslug (e.g. `openai/gpt-4o`), not the display name. - Other dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, etc.) are not enriched — filter values match what's returned in results. @@ -190,7 +191,7 @@ Classifier filters narrow results to generations matching specific classificatio > **Numeric types:** Count metrics (`request_count`, `tokens_*`, etc.) are returned as strings (`"1523"`). Cost and rate metrics (`total_usage`, `cache_hit_rate`, latency, throughput) are returned as numbers (`4.27`). Parse count values with `Number()` or `parseInt()` before arithmetic. -> **Label resolution:** Dimensions `api_key_id`, `app`, `user`, and `workspace` return human-readable labels in data rows (key names, app titles, user names, workspace names), not raw IDs. +> **Label resolution:** Dimensions `api_key_id`, `app`, `user`, and `workspace` return human-readable labels in data rows (key names, app titles, user names, workspace names), not raw IDs. The unset buckets `api_key_id = -1` and `app = -1` resolve to `Chatroom` and `Unknown`. The unresolved-workspace sentinel `ffffffff-ffff-ffff-ffff-ffffffffffff` resolves to `Unattributed` (a different value from the legacy all-zero UUID, which is folded into the default workspace). ## CLI Reference @@ -339,5 +340,5 @@ Classifier dimensions and classifier filters always force the 31-day time range If a query times out, try: - Narrowing the time range - Removing latency/throughput metrics -- Removing per-generation dimensions (`provider`, `origin`, `country`, `finish_reason`, etc.) +- Removing per-generation dimensions (`provider`, `origin`, `country`, `data_region`, `finish_reason`, etc.) - Removing classifier dimensions/filters (they are more expensive to compute) diff --git a/skills/openrouter-analytics-schema/SKILL.md b/skills/openrouter-analytics-schema/SKILL.md index d05608b..a263a2f 100644 --- a/skills/openrouter-analytics-schema/SKILL.md +++ b/skills/openrouter-analytics-schema/SKILL.md @@ -60,7 +60,7 @@ Each metric has: ### Time Range Limits -Most volume and cost metrics support time ranges up to **365 days** with daily granularity. Latency/throughput metrics and some dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, `context_length_bucket`, `generation_id`) are limited to **31-day** time ranges. If a query times out, try narrowing the time range or removing latency/throughput metrics and per-generation dimensions. +Most volume and cost metrics support time ranges up to **365 days** with daily granularity. Latency/throughput metrics and some dimensions (`provider`, `origin`, `country`, `data_region`, `finish_reason`, `external_user`, `context_length_bucket`, `generation_id`, `session_id`) are limited to **31-day** time ranges. If a query times out, try narrowing the time range or removing latency/throughput metrics and per-generation dimensions. ### Metric Categories @@ -95,6 +95,7 @@ Most volume and cost metrics support time ranges up to **365 days** with daily g **Efficiency metrics** (how well): - `cache_hit_rate` — ratio of cached tokens to prompt tokens (0–1) +- `blended_cost_per_million_tokens` — total spend, including BYOK inference, divided by prompt plus completion tokens and multiplied by 1M. Currency rate metric in USD per 1M tokens (up to 365 days). NULL when there are no tokens. - `guardrail_invoked_rate` — ratio of requests that triggered guardrails - `response_cached_rate` — ratio of responses served from cache @@ -122,7 +123,7 @@ Some dimensions have their raw IDs automatically resolved to human-readable labe | `user` | User name or email address | | `workspace` | Workspace name | -All other dimensions (e.g., `model`, `provider`, `country`) are returned as-is without resolution. +The `api_key_id` value `-1` resolves to `Chatroom`, `app` value `-1` resolves to `Unknown`, and the unresolved-workspace sentinel `ffffffff-ffff-ffff-ffff-ffffffffffff` resolves to `Unattributed` (distinct from the legacy all-zero UUID, which is folded into the default workspace). All other dimensions (e.g., `model`, `provider`, `country`) are returned as-is without resolution. > Rows with an empty `user` value represent traffic not attributed to a specific org member (e.g., API keys created at the org level). @@ -133,7 +134,7 @@ All other dimensions (e.g., `model`, `provider`, `country`) are returned as-is w - `variant` — model variant (e.g., standard, extended) - `api_key_id` — which API key made the request - `user` — the creator user ID (for org-level queries) -- `workspace` — workspace ID +- `workspace` — workspace ID. Filtering or grouping by the account default workspace also covers activity recorded before workspace resolution existed, which is folded from the legacy all-zero UUID. Grouping does not return the all-zero UUID as a separate group. - `app` — application ID **Limited to 31-day time ranges:** @@ -141,9 +142,11 @@ All other dimensions (e.g., `model`, `provider`, `country`) are returned as-is w - `provider` — upstream provider name - `origin` — request origin/source - `country` — request country +- `data_region` — request data region (`global`, `europe`, or `us`). Rows predating the default are reported as `global`. - `finish_reason` — why the generation ended (stop, length, etc.) - `external_user` — custom user ID passed by the caller - `context_length_bucket` — bucketed context length (1K, 10K, 100K, etc.) +- `session_id` — session ID. Sessionless requests are returned as the literal `none`. ## Classifier Dimensions @@ -178,6 +181,8 @@ Filter operators for the `filters` array in query requests: | `in` | array | In list | | `not_in` | array | Not in list | +The optional `include_unset` flag is supported only with `in` and `not_in`. It is valid for `api_key_id` with sentinel `-1`, `app` with sentinel `-1`, and `user` with sentinel empty string. With `in`, it adds unset rows to the matched set. With `not_in`, unset rows are excluded along with the listed values. An empty array with `in` matches only unset rows. Other dimensions fail validation with `Dimension "" has no unset bucket`. The `/meta` response does not indicate which dimensions have an unset bucket. + ## Understanding Granularities Time bucketing for time-series queries: @@ -204,6 +209,7 @@ Use this guide to translate natural-language questions into the right metric/dim | "How many tokens?" | `tokens_total` | — | Use `tokens_prompt` / `tokens_completion` for split | | "Which provider is fastest?" | `avg_latency`, `p90_latency` | `provider` | 31-day limit | | "What's my cache hit rate?" | `cache_hit_rate` | `model` | Rate metric — shows per-model caching | +| "What's my effective cost per million tokens?" | `blended_cost_per_million_tokens` | `model` | Rate metric in USD per 1M tokens. Includes BYOK inference spend and is NULL when there are no tokens. | | "Which API key uses the most?" | `request_count`, `total_usage` | `api_key_id` | — | | "Usage over time" | `request_count` or `total_usage` | — | Set `granularity: "day"` | | "Latency trends" | `p90_latency` | — | Set `granularity: "hour"`, 31d limit | @@ -229,7 +235,7 @@ Several dimensions are **label-resolved** in query results — the response show |---|---|---| | `api_key_id` | Numeric ID **or** 64-char SHA-256 hash | Numeric ID: generation metadata (`api_key_id` field). Hash: `GET /api/v1/keys` (`key_hash` field). Hashes are auto-resolved server-side. If a hash can't be resolved, a sentinel value returns zero rows (no error). | | `user` | Clerk user ID (e.g. `user_abc123`) | User settings or org member list — not the display name/email shown in results. | -| `workspace` | Workspace UUID | Workspace settings page or `GET /api/v1/workspaces` — not the workspace name shown in results. | +| `workspace` | Workspace UUID | Workspace settings page or `GET /api/v1/workspaces` — not the workspace name shown in results. The account default workspace also matches legacy all-zero UUID rows, which are folded into that workspace for grouping. | | `app` | Numeric app ID | Generation metadata (`app_id` field) or app settings — not the app title shown in results. | | `model` | Permaslug (e.g. `openai/gpt-4o`) | Model page URL or `GET /api/v1/models` — not the display name. | diff --git a/skills/openrouter-analytics/SKILL.md b/skills/openrouter-analytics/SKILL.md index 2b7cdd3..fc7c62b 100644 --- a/skills/openrouter-analytics/SKILL.md +++ b/skills/openrouter-analytics/SKILL.md @@ -127,7 +127,7 @@ When interpreting results for the user: - **Rates** (`cache_hit_rate`) are 0–1 ratios - **Throughput** (`avg_throughput`) is tokens per second - When `granularity` is set, rows include a `date__` field for the time bucket (e.g., `date__day`, `date__hour`, `date__month`) -- **Label resolution**: dimensions `api_key_id`, `app`, `user`, and `workspace` have their raw IDs replaced with human-readable names (key name, app title, user name, workspace name) directly in the data rows; `generation_id` and `session_id` return raw values +- **Label resolution**: dimensions `api_key_id`, `app`, `user`, and `workspace` have their raw IDs replaced with human-readable names (key name, app title, user name, workspace name) directly in the data rows; `api_key_id = -1` resolves to `Chatroom`, `app = -1` resolves to `Unknown`, and the unresolved-workspace sentinel `ffffffff-ffff-ffff-ffff-ffffffffffff` resolves to `Unattributed` (distinct from the legacy all-zero UUID, which is folded into the default workspace); `generation_id` and `session_id` return raw values - **Truncation**: when consuming output programmatically, check `metadata.truncated`. If `true`, the result was capped at `--limit` and is a *partial* dataset — raise `--limit` or paginate before reporting totals or rankings ### Cost Optimization Guidance