Add lemonade-liquid-router: local-first tri-ensemble router (~80% on-device)

[ramkrishna2910]

Summary

lemonade-liquid-router is a local-first router built on three on-device models — two generator voters and an embedding referee — all served by a single instance of Lemonade, AMD's open-source local LLM server, plus one cloud voter. ~80% of the benchmark is answered entirely on-device.

Local scoring with this repo's evaluation code: accuracy 78.10% · $0.178 per 1K queries · robustness 73.6.

Candidate models

Model	Runs	Role
lemonade/deepseek-v4-flash	Local	Primary voter, preferred response source
lemonade/Qwen3.8-27B-GGUF-UD-Q4_K_XL	Local	Second voter, corroborator
lemonade/LFM2.5-Embedding-350M	Local	Semantic referee: embedding-cosine veto on degraded free-answer comparisons
gemini-3-flash-preview	Cloud	Third voter, escalation target, code specialist
deepseek/deepseek-v4-pro	Cloud	Declared escalation fallback (unused — every escalation resolved to gemini-3-flash-preview)

The local models are served by a single Lemonade instance: DeepSeek-V4-Flash as an IQ2XXS quant via the ds4 recipe, Qwen3.8-27B as UD-Q4_K_XL and LFM2.5-Embedding-350M as an F16 GGUF via the llamacpp recipe. Hardware: 2× AMD Ryzen AI Max+ 395 (128 GB unified memory each).

How the router works

Each query is classified by prompt shape alone, then routed:

1. Code (no Options: block, no boxed-answer instruction) → answered by gemini-3-flash-preview directly.
2. MCQ (Options: present) → all three voters answer independently; the boxed letter is extracted from each response and a 2-of-3 majority vote is taken. The submitted response comes from a majority member, preferring local models (DeepSeek, then Qwen, then gemini). No majority → escalate to gemini-3-flash-preview.
3. Free-answer (everything else) → both local models answer; the final answers are extracted (boxed content when present) and compared. If Qwen corroborates DeepSeek — exact match or token-F1 ≥ 0.5 on the extracted answers — the local DeepSeek answer is submitted. When either side produced no boxed answer, the comparison ran on degraded last-200-characters extracts; keeping the local answer then additionally requires the on-device embedding judge (lemonade/LFM2.5-Embedding-350M) to corroborate semantically: cosine ≥ 0.7 between the two extracts. Any failed check → escalate to gemini-3-flash-preview.

What the routing is based on

Routing uses only the prompt text and the models' own outputs (agreement between voters, measured lexically and — on degraded extracts — by embedding cosine from an off-the-shelf embedding model). There is no routing model trained by us, no query-difficulty classifier, and no per-domain rules.

The design constants — the 2-of-3 vote, the token-F1 ≥ 0.5 corroboration threshold, the embedding-veto threshold (cosine ≥ 0.7, chosen label-free at the precision knee of a g3fp-agreement pseudo-reference), the local-preference order — were fixed a priori and calibrated exclusively on a 1,251-item set sampled from the upstream public benchmarks (deduplicated against all RouterArena splits) and on our own runtime/token telemetry. Model selection (which model fills the cloud slot) was additionally informed by aggregate statistics from the graded submissions already public in this repo — the same information available to every submitter, never used to fit any per-query component. No RouterArena data or labels were used to tune any router component. Full-set labels were touched only by pre-registered scoring runs of frozen configurations.

Reproducing the results

Prerequisites: a machine that can serve the three local models, OPENROUTER_API_KEY (or GOOGLE_API_KEY for direct Gemini access), and LEMONADE_BASE_URL if the Lemonade server is not on the default http://localhost:8000/api/v1.

# 1. Serve the three local models with the public Lemonade release:
#    deepseek-v4-flash (ds4 recipe), Qwen3.8-27B-GGUF-UD-Q4_K_XL (llamacpp recipe),
#    and the embedding judge:
lemonade pull user.LFM2.5-Embedding-350M --recipe llamacpp \
  --checkpoint main "LiquidAI/LFM2.5-Embedding-350M-GGUF:LFM2.5-Embedding-350M-F16.gguf" \
  --label embeddings
lemonade-server serve

# 2. Standard RouterArena pipeline
uv run python ./scripts/process_datasets/prep_datasets.py
uv run python ./router_inference/generate_prediction_file.py lemonade-liquid-router full
uv run python ./router_inference/check_config_prediction_files.py lemonade-liquid-router full
uv run python ./llm_inference/run.py lemonade-liquid-router
uv run python ./llm_evaluation/run.py lemonade-liquid-router full
uv run python ./llm_evaluation/run.py lemonade-liquid-router robustness

The local side is fully deterministic (temperature 0, thinking disabled, max_tokens 4096), and we verified the local IQ2XXS DeepSeek-V4-Flash quant scores identically to the API model on our calibration set, so the numbers should reproduce up to cloud-provider nondeterminism.

Full methodology — the frozen policy as run, the provenance of every design constant, and the calibration set builder with a per-item hash manifest so the dedup claim is independently checkable: https://github.com/ramkrishna2910/lemonade-router-routerarena. Raw generation logs contain RouterArena prompt content and are therefore not republished; they are available to maintainers on request.

Pricing & transparency

• Cloud models use their existing entries / public list prices.
• The lemonade/* local models are declared at an honest self-hosting estimate (power + amortization): $0.02/$0.04 per Mtok in/out for DeepSeek-V4-Flash, $0.01/$0.02 for Qwen3.8-27B, $0.001/$0.001 for the embedding judge (which never appears as a prediction; its entry exists for config completeness).
• The submitted cost reflects the tokens of each submitted response. For transparency: the router additionally spends local (free) tokens and one cloud vote per MCQ query at serving time.

Files

• router_inference/config/lemonade-liquid-router.json
• router_inference/predictions/lemonade-liquid-router.json — 8,400 rows, generated_result populated
• router_inference/predictions/lemonade-liquid-router-robustness.json — 420 rows
• router_inference/router/lemonade_liquid_router.py + registry entry
• llm_inference/model_inference.py — lemonade (local OpenAI-compatible) and OpenRouter provider mappings
• universal_model_names.py, model_cost/model_cost.json — entries for the lemonade/* models

Both validation scripts pass (check_config_prediction_files.py for full and robustness), and the full pre-commit suite — including the CI mypy matrix (3.9–3.12) — passes on the branch.

Leaderboard row details (to save you a lookup)

• Router: lemonade-liquid-router
• Affiliation: Lemonade (AMD's open-source local LLM server)
• Links: Methodology & calibration artifacts · Lemonade
• Contact: @ramkrishna2910

🤖 Generated with Claude Code

[ramkrishna2910]

/evaluate

[github-actions]

Router Evaluation Results

Router: lemonade-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7670
Accuracy	78.04%
Total Cost	$1.497923
Avg Cost per Query	$0.000178
Avg Cost per 1K Queries	$0.1783
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7357

---

Evaluation completed by RouterArena automated workflow

[ramkrishna2910]

/evaluate

[github-actions]

Router Evaluation Results

Router: lemonade-liquid-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7670
Accuracy	78.04%
Total Cost	$1.497923
Avg Cost per Query	$0.000178
Avg Cost per 1K Queries	$0.1783
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7357

---

Evaluation completed by RouterArena automated workflow

[yl231]

Thanks for an unusually thorough and transparent submission, @ramkrishna2910 — the methodology write-up, the frozen-config provenance, and the dedup manifest are exactly what we like to see, and I want to be clear up front that nothing here reads as bad-faith. The integrity audit is clean: 8,400 real rows, genuine per-prompt token accounting, no oracle/label contamination, faithfulness in range. The routing logic is real.

I'm holding the leaderboard placement, though, on cost accounting — two issues, one policy and one concrete:

1. Uncharged voter tokens (concrete). You disclose that every MCQ query runs all three voters — including a cloud gemini-3-flash-preview vote — and free-answer queries run both local models, but the submitted cost counts only the tokens of the single winning response. RouterArena's cost axis is meant to capture everything the router spends to make its decision, not just the returned answer. As a reference point, the cruq-router submission (#197) charges for all K=4 of its self-consistency probes. Measured that way, lemonade-liquid is paying real cloud (gemini) tokens on every MCQ that currently aren't billed at all, plus the second local voter — so the reported $0.178/1K is understated regardless of how the local models are priced.

2. Local/self-hosted pricing (policy — see #193). The lemonade/deepseek-v4-flash entry is $0.02/$0.04 per Mtok, a self-hosting power+amortization estimate, while the same-named model already sits on the board (cloud deepseek/deepseek-v4-flash) at $0.14/$0.28. We're settling the general rule in #193, but the working position is that a model is priced at its standard published list price regardless of the endpoint used, precisely because self-declared on-device estimates aren't externally verifiable or comparable across submitters. If you want to argue that a genuinely-quantized local variant should be treated as a distinct, differently-priced model, #193 is the thread to make that case.

What would unblock it: re-account the cost so that every model call the router makes per query is charged (all voters, including the cloud gemini vote), and align the local pricing with the #193 outcome. I expect (1) alone moves the number materially. The router itself is fine — this is purely about making the cost comparable to the cloud-based entries. Happy to re-review as soon as the accounting is updated, and thanks again for the care you put into this.

[ramkrishna2910]

Thanks for the careful review and kind words, you’re right on point 1. We disclosed the extra voter calls but billed only the winning response’s tokens, we’ll fix that.

One property of the policy defines what needs to be charged: in a 2-of-3 majority vote, the cloud vote cannot change the outcome when the two local voters agree — the majority is already decided. The locals agree on 81.2% of MCQs (4,810 of 5,924). We will re-account by charging gemini vote on the 1,114 MCQs where the two locals disagree (plus all escalations, which are already billed as submitted rows). This short-circuit implementation produces the identical prediction file, and we’ll update the router class to match. Re-accounted cost: $0.249/1K.

On point 2, we’ll make the case in #193. Briefly: lemonade/deepseek-v4-flash is not the cloud model at a different endpoint, it’s a distinct public artifact (2-bit IQ2XXS quant, weights on HF), served by public Lemonade release. Anyone can reproduce both the quality and the cost. This submission exists to show local models augmenting cloud usage; pricing on-device inference at cloud list rates charges dollars no one spends and erases the deployment mode it demonstrates.

We’ll push the re-accounted files if the above makes sense. Looking forward to your response :)