Skip to content

fix(pricing): resolve model names canonically so priced models stop falling through - #201

Open
ypollak2 wants to merge 1 commit into
RouteWorks:mainfrom
ypollak2:fix/canonical-model-key
Open

fix(pricing): resolve model names canonically so priced models stop falling through#201
ypollak2 wants to merge 1 commit into
RouteWorks:mainfrom
ypollak2:fix/canonical-model-key

Conversation

@ypollak2

Copy link
Copy Markdown
Contributor

Follow-up to #193, implementing the normalizer half of what was discussed there.

The problem

Pricing a row is an exact dict lookup of the submitter-written model name against model_cost/model_cost.json. The name and the key have to match character for character, so a model that is genuinely in the price table can still fail to resolve — on letter case, or on _ vs / as the vendor separator. Once it fails to resolve, the row can't be charged.

Two of these are live on the board right now:

router model named rows share why it fails
lynkr openai/gpt-oss-120b 8,096 96.4% priced, but only as openai_gpt-oss-120b
glm-4-air-router glm-4-air 8,400 100% no price entry under any spelling

The first is a spelling mismatch on a model that is priced — nothing about it should be a pricing question. The second is the case #193's policy is actually for.

The registry disagrees with itself in the same two ways: of 97 keys, 92 are lowercase and 5 aren't; separators split 54 bare / 30 vendor/model / 13 vendor_model. So this will keep happening as long as the lookup is exact.

What this changes

universal_model_names.py gains canonical_key() and resolve_universal_name(). The fold normalises case and the vendor separator and nothing else — vendor prefixes are deliberately not stripped, because vendor/foo and a bare foo are not reliably the same model, and the existing mapping table already records the cases where they are.

Both ModelNameManager lookups now route through it, with resolution order unchanged: exact universal name → explicit mapping → canonical fold.

No existing caller changes behaviour. get_universal_name still raises ValueError on an unresolved name; get_universal_name_non_static still records it in missing_models and returns it unchanged rather than raising. I checked all 51 distinct model names across every prediction file in the repo: 0 resolve differently than before, and missing_models comes back empty. This PR strictly widens what resolves; it never redirects a name that already worked.

I left the non-static variant non-fatal on purpose. Making it raise is a policy call about what happens to entries already on the board, which is the open question in #193 — worth deciding there, not smuggled into a normalizer PR. The audit tool below is the fail-loud gate in the meantime, and is CI-ready.

tools/check_model_names.py

Follows tools/audit_token_accounting.py in shape (SPDX header, per-router table, --strict exit code). Current output on main:

Problem 1 - price-table keys that differ only by case or separator
  [PRICE CONFLICT] qwen/qwen3-235b-a22b-2507
      keys:   ['qwen/qwen3-235b-a22b-2507', 'qwen_qwen3-235b-a22b-2507']
      prices: [(0.071, 0.1), (0.071, 0.463)]
  [duplicate] z-ai/glm-4.7
      keys:   ['z-ai/glm-4.7', 'z-ai_glm-4.7']
      prices: [(0.4, 1.5)]

Problem 2 - universal names that fold together ambiguously
  google/gemini-2.5-flash-lite <- ['gemini-2.5-flash-lite', 'google/gemini-2.5-flash-lite']

Problem 3 - models named by a submission that cannot be priced
router                     model named                                rows   share  resolution
----------------------------------------------------------------------------------------------
glm-4-air-router           glm-4-air                                  8400  100.0%  NO PRICE ENTRY - register one (issue #193 policy)
lynkr                      openai/gpt-oss-120b                        8096   96.4%  spelling only - folds to openai_gpt-oss-120b

Three findings there are worth maintainer attention independently of this PR:

  1. qwen/qwen3-235b-a22b-2507 is registered twice at two different prices$0.071/$0.100 under the slash spelling, $0.071/$0.463 under the underscore spelling. Same model, 4.6× difference in output price, selectable by spelling. Sixteen routers on the board name the slash form. This is exactly the relabel-for-a-cheaper-rate hole Policy for models absent from model_cost.json, and for free/promotional endpoints #193 sets out to close, except it needs no relabeling — just a choice of which existing key to type. I did not pick a winner here; that's the reference-price decision from Policy for models absent from model_cost.json, and for free/promotional endpoints #193 and it should be made deliberately.

  2. z-ai/glm-4.7 is duplicated too, but at identical prices — harmless today, same failure waiting.

  3. google/gemini-2.5-flash-lite is both an entry in universal_names and a key in mapping pointing at the different name gemini-2.5-flash-lite. Which one you get depends on lookup order. canonical_collisions surfaces this rather than raising at import, since importing this module should never be the thing that breaks a run.

I deliberately kept the reference-price precedence rule and the glm-4-air price itself out of this PR — both are policy calls for #193, and this is just the mechanism they'd need. Happy to follow up with either once the thread lands.

…alling through

A model named by a prediction file is priced by an exact lookup in
model_cost.json. The name is written by the submitter, so a model that IS in
the price table can still fail to resolve on letter case or on the vendor
separator, and an unpriceable row is then scored around rather than charged.

Two such names are live on the board today:

  lynkr            openai/gpt-oss-120b   8096 rows (96.4%)
                   priced only as openai_gpt-oss-120b
  glm-4-air-router glm-4-air             8400 rows (100%)
                   no price entry under any spelling

Add canonical_key()/resolve_universal_name() to universal_model_names.py,
folding case and the `_` vs `/` vendor separator and nothing else -- vendor
prefixes are deliberately not stripped, since `vendor/foo` and a bare `foo`
are not reliably the same model and `mapping` already records the cases where
they are. Both ModelNameManager lookups now go through it; the static variant
still raises on an unresolved name, and the non-static variant still records
it in missing_models rather than raising, so no existing caller changes
behaviour. Verified: all 51 distinct model names across every prediction file
resolve exactly as before.

Add tools/check_model_names.py, following tools/audit_token_accounting.py, to
report the three failure modes and exit 1 under --strict when a named model
has no price entry at all.

Refs RouteWorks#193

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BydrpgVJTHVLVkCZT9wMXq
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant