fix(pricing): resolve model names canonically so priced models stop falling through - #201
Open
ypollak2 wants to merge 1 commit into
Open
fix(pricing): resolve model names canonically so priced models stop falling through#201ypollak2 wants to merge 1 commit into
ypollak2 wants to merge 1 commit into
Conversation
…alling through
A model named by a prediction file is priced by an exact lookup in
model_cost.json. The name is written by the submitter, so a model that IS in
the price table can still fail to resolve on letter case or on the vendor
separator, and an unpriceable row is then scored around rather than charged.
Two such names are live on the board today:
lynkr openai/gpt-oss-120b 8096 rows (96.4%)
priced only as openai_gpt-oss-120b
glm-4-air-router glm-4-air 8400 rows (100%)
no price entry under any spelling
Add canonical_key()/resolve_universal_name() to universal_model_names.py,
folding case and the `_` vs `/` vendor separator and nothing else -- vendor
prefixes are deliberately not stripped, since `vendor/foo` and a bare `foo`
are not reliably the same model and `mapping` already records the cases where
they are. Both ModelNameManager lookups now go through it; the static variant
still raises on an unresolved name, and the non-static variant still records
it in missing_models rather than raising, so no existing caller changes
behaviour. Verified: all 51 distinct model names across every prediction file
resolve exactly as before.
Add tools/check_model_names.py, following tools/audit_token_accounting.py, to
report the three failure modes and exit 1 under --strict when a named model
has no price entry at all.
Refs RouteWorks#193
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BydrpgVJTHVLVkCZT9wMXq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #193, implementing the normalizer half of what was discussed there.
The problem
Pricing a row is an exact dict lookup of the submitter-written model name against
model_cost/model_cost.json. The name and the key have to match character for character, so a model that is genuinely in the price table can still fail to resolve — on letter case, or on_vs/as the vendor separator. Once it fails to resolve, the row can't be charged.Two of these are live on the board right now:
lynkropenai/gpt-oss-120bopenai_gpt-oss-120bglm-4-air-routerglm-4-airThe first is a spelling mismatch on a model that is priced — nothing about it should be a pricing question. The second is the case #193's policy is actually for.
The registry disagrees with itself in the same two ways: of 97 keys, 92 are lowercase and 5 aren't; separators split 54 bare / 30
vendor/model/ 13vendor_model. So this will keep happening as long as the lookup is exact.What this changes
universal_model_names.pygainscanonical_key()andresolve_universal_name(). The fold normalises case and the vendor separator and nothing else — vendor prefixes are deliberately not stripped, becausevendor/fooand a barefooare not reliably the same model, and the existingmappingtable already records the cases where they are.Both
ModelNameManagerlookups now route through it, with resolution order unchanged: exact universal name → explicitmapping→ canonical fold.No existing caller changes behaviour.
get_universal_namestill raisesValueErroron an unresolved name;get_universal_name_non_staticstill records it inmissing_modelsand returns it unchanged rather than raising. I checked all 51 distinct model names across every prediction file in the repo: 0 resolve differently than before, andmissing_modelscomes back empty. This PR strictly widens what resolves; it never redirects a name that already worked.I left the non-static variant non-fatal on purpose. Making it raise is a policy call about what happens to entries already on the board, which is the open question in #193 — worth deciding there, not smuggled into a normalizer PR. The audit tool below is the fail-loud gate in the meantime, and is CI-ready.
tools/check_model_names.pyFollows
tools/audit_token_accounting.pyin shape (SPDX header, per-router table,--strictexit code). Current output onmain:Three findings there are worth maintainer attention independently of this PR:
qwen/qwen3-235b-a22b-2507is registered twice at two different prices —$0.071/$0.100under the slash spelling,$0.071/$0.463under the underscore spelling. Same model, 4.6× difference in output price, selectable by spelling. Sixteen routers on the board name the slash form. This is exactly the relabel-for-a-cheaper-rate hole Policy for models absent from model_cost.json, and for free/promotional endpoints #193 sets out to close, except it needs no relabeling — just a choice of which existing key to type. I did not pick a winner here; that's the reference-price decision from Policy for models absent from model_cost.json, and for free/promotional endpoints #193 and it should be made deliberately.z-ai/glm-4.7is duplicated too, but at identical prices — harmless today, same failure waiting.google/gemini-2.5-flash-liteis both an entry inuniversal_namesand a key inmappingpointing at the different namegemini-2.5-flash-lite. Which one you get depends on lookup order.canonical_collisionssurfaces this rather than raising at import, since importing this module should never be the thing that breaks a run.I deliberately kept the reference-price precedence rule and the
glm-4-airprice itself out of this PR — both are policy calls for #193, and this is just the mechanism they'd need. Happy to follow up with either once the thread lands.