Skip to content

docs(openrouter-benchmarks): sync with openrouter-web changes #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
devin-ai-integration wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(openrouter-benchmarks): sync with openrouter-web changes #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions skills/openrouter-benchmarks/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
name: openrouter-benchmarks
description: Query OpenRouter's Benchmarks API for model benchmark rankings and scores. Use when the user asks for benchmark-backed model selection, model rankings by coding/intelligence/agentic ability, Artificial Analysis or Design Arena ELO/win-rate results, benchmark citations, or wants to call GET /api/v1/benchmarks. Also use alongside openrouter-models when the user asks what model should power an app, product, workflow, or use case and benchmark evidence could inform or rule out part of the recommendation, including creative writing, editing, coding, design, agentic, or intelligence-heavy apps. Do not use for OpenRouter usage analytics, billing/spend analysis, generation metadata, provider uptime/latency, generic model pricing/capability lookup without any selection or benchmark-relevance decision, or creating an evaluation suite for a local app.
description: Query OpenRouter's Benchmarks API for model benchmark rankings and scores. Use when the user asks for benchmark-backed model selection, model rankings by coding/intelligence/agentic ability, Artificial Analysis or Design Arena ELO/win-rate results, OpenRouter GPQA or tau-bench results, benchmark citations, or wants to call GET /api/v1/benchmarks. Also use alongside openrouter-models when the user asks what model should power an app, product, workflow, or use case and benchmark evidence could inform or rule out part of the recommendation, including creative writing, editing, coding, design, agentic, or intelligence-heavy apps. Do not use for OpenRouter usage analytics, billing/spend analysis, generation metadata, provider uptime/latency, generic model pricing/capability lookup without any selection or benchmark-relevance decision, or creating an evaluation suite for a local app.
---

# OpenRouter Benchmarks

Use OpenRouter's unified benchmarks endpoint to answer benchmark-backed model ranking and model-selection questions. The endpoint aggregates Artificial Analysis and Design Arena data and returns citation metadata that should be preserved when reporting results.
Use OpenRouter's unified benchmarks endpoint to answer benchmark-backed model ranking and model-selection questions. The endpoint aggregates Artificial Analysis, Design Arena, and OpenRouter's own GPQA and tau-bench data and returns citation metadata that should be preserved when reporting results.

## Prerequisites

Expand All @@ -24,6 +24,7 @@ export OPENROUTER_API_KEY=sk-or-v1-...
| Find best coding, intelligence, or agentic models | Use `task_type=coding`, `task_type=intelligence`, or `task_type=agentic` |
| Query Artificial Analysis only | Use `source=artificial-analysis` |
| Query Design Arena only | Use `source=design-arena`, plus `arena` and `category` when relevant |
| Query OpenRouter GPQA or tau-bench results | Use `source=openrouter`; use `task_type=intelligence` for GPQA or `task_type=agentic` for tau-bench |
| Get raw API-shaped data for integration work | Return the raw `data`/`meta` shape from the endpoint |
| Understand all response fields or direct curl usage | Read `references/benchmarks-api.md` |

Expand Down Expand Up @@ -51,21 +52,22 @@ Query parameters:

| Flag | Values | Notes |
|---|---|---|
| `source` | `artificial-analysis`, `design-arena` | Omitting it returns all sources. |
| `source` | `artificial-analysis`, `design-arena`, `openrouter` | Omitting it returns all sources. |
| `task_type` | `coding`, `intelligence`, `agentic` | Maps to source-specific indices/categories. |
| `arena` | `models`, `builders`, `agents` | Design Arena only; defaults server-side to `models`. |
| `category` | `codecategories`, `uicomponent`, `gamedev`, `3d`, `dataviz`, `image`, `video`, `svg`, etc. | Design Arena only. |
| `max_results` | positive integer | Maximum number of rows returned by the API. |

Always preserve `meta.citation`, `meta.source_url`, and `meta.as_of`; include attribution when republishing benchmark data.

When results include both sources, do not present them as a single absolute leaderboard: Artificial Analysis indices and Design Arena ELO use different scales. Compare within each source, or rerun with `source=artificial-analysis` or `source=design-arena` for a source-specific ranking.
When results include multiple sources, do not present them as a single absolute leaderboard: Artificial Analysis indices, Design Arena ELO, and OpenRouter accuracy scores use different scales and evaluation contracts. Compare within each source, or rerun with `source=artificial-analysis`, `source=design-arena`, or `source=openrouter` for a source-specific ranking.

## Interpreting Results

- Artificial Analysis rows include `intelligence_index`, `coding_index`, and `agentic_index`; higher is better.
- Design Arena rows include `elo`, `win_rate`, `avg_generation_time_ms`, `arena`, `category`, and `tournament_stats`; higher `elo`/`win_rate` is better, lower generation time is faster.
- `pricing.prompt` and `pricing.completion` are USD per token as decimal strings. Multiply by 1,000,000 for per-million-token costs.
- OpenRouter rows include `benchmark_type`, `accuracy`, `accuracy_stddev`, `avg_cost_per_task`, `total_tasks`, and `last_run_timestamp`; `accuracy` is a 0–1 score where higher is better. OpenRouter benchmark types are `gpqa_diamond` and `tau_bench_verified_airline`.
- `pricing.prompt` and `pricing.completion` on Artificial Analysis and Design Arena rows are USD per token as decimal strings. Multiply by 1,000,000 for per-million-token costs; OpenRouter rows do not include a `pricing` field.
- `model_permaslug` identifies the benchmarked model entry. Verify it against `GET /api/v1/models` before using it as a chat/completions model ID.
- `meta.model_count` counts unique models in the response, which can differ from `data.length` when multiple Design Arena categories are returned.

Expand Down
30 changes: 24 additions & 6 deletions skills/openrouter-benchmarks/references/benchmarks-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ GET https://openrouter.ai/api/v1/benchmarks
Authorization: Bearer <OPENROUTER_API_KEY>
```

The endpoint aggregates Artificial Analysis and Design Arena benchmark scores. It is authenticated with any valid OpenRouter API key and rate-limited to 30 requests/minute per key and 500 requests/day per account.
The endpoint aggregates Artificial Analysis, Design Arena, and OpenRouter's own GPQA and tau-bench benchmark scores. It is authenticated with any valid OpenRouter API key and rate-limited to 30 requests/minute per key and 500 requests/day per account.

## Query Parameters

| Parameter | Values | Description |
|---|---|---|
| `source` | `artificial-analysis`, `design-arena` | Benchmark source. Omitting it returns all sources. The source determines row shape. |
| `task_type` | `coding`, `intelligence`, `agentic` | Workload filter. For Artificial Analysis, maps to the corresponding index. For Design Arena, maps to the matching category. |
| `source` | `artificial-analysis`, `design-arena`, `openrouter` | Benchmark source. Omitting it returns all sources. The source determines row shape. |
| `task_type` | `coding`, `intelligence`, `agentic` | Workload filter. For Artificial Analysis, maps to the corresponding index. For Design Arena, maps to the matching category. For OpenRouter, `intelligence` maps to `gpqa_diamond` and `agentic` maps to `tau_bench_verified_airline`. |
| `arena` | `models`, `builders`, `agents` | Design Arena only. Defaults to `models` when `source=design-arena`. |
| `category` | string | Design Arena category such as `codecategories`, `uicomponent`, `gamedev`, `3d`, `dataviz`, `image`, `video`, or `svg`. Omitting it returns all categories. |
| `max_results` | integer >= 1 | Maximum number of items to return. Omitting it returns all matching results. |
Expand All @@ -25,12 +25,12 @@ The endpoint aggregates Artificial Analysis and Design Arena benchmark scores. I

```ts
type UnifiedBenchmarksResponse = {
data: Array<ArtificialAnalysisItem | DesignArenaItem>;
data: Array<ArtificialAnalysisItem | DesignArenaItem | OpenRouterItem>;
meta: {
as_of: string;
citation: string | null;
model_count: number;
source: "artificial-analysis" | "design-arena" | null;
source: "artificial-analysis" | "design-arena" | "openrouter" | null;
source_url: string | null;
task_type: string | null;
version: "v1";
Expand Down Expand Up @@ -79,6 +79,24 @@ type DesignArenaItem = {

Higher `elo` and `win_rate` are better. `avg_generation_time_ms` is performance context, not the ranking score.

### OpenRouter Item

```ts
type OpenRouterItem = {
source: "openrouter";
model_permaslug: string;
display_name: string;
benchmark_type: "gpqa_diamond" | "tau_bench_verified_airline";
accuracy: number;
accuracy_stddev: number | null;
avg_cost_per_task: number | null;
total_tasks: number;
last_run_timestamp: string;
};
```

`accuracy` is an aggregate score from 0 to 1; higher is better. OpenRouter rows do not include a `pricing` field.

## Errors

| Status | Meaning | Recovery |
Expand All @@ -90,7 +108,7 @@ Higher `elo` and `win_rate` are better. `avg_generation_time_ms` is performance

## Reporting Guidance

When answering users, include the benchmark source, `meta.as_of`, and the citation/source URL if present. If results mix sources and `meta.citation` is null, attribute each row by its `source` discriminator.
When answering users, include the benchmark source, `meta.as_of`, and the citation/source URL if present. For `source=openrouter`, `meta.citation` is `Source: OpenRouter evals (openrouter.ai) via OpenRouter (openrouter.ai/rankings).` and `meta.source_url` is `https://openrouter.ai/rankings`. If results mix sources and `meta.citation` is null, attribute each row by its `source` discriminator.

## Availability Caveat

Expand Down