Submit R**2: Reasonometry Router - #191
Conversation
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
|
Thanks for the submission, @sashakolpakov. Maintainer review — I can't accept the current result because the cost is not real, and the cost axis is what places this at the top of the board. Auditing
Hard-coded token counts produce the reported $0.04/1K, which is what yields the #1 Arena score — so the ranking is an artifact of the fabricated metering, not the router. This also contradicts the PR's own description of a "multi-round / API-calling reasoning router." To be reconsidered, please resubmit with real token accounting from the actual model calls (true per-prompt input tokens and full completion output), and predictions that reflect the multi-round behavior you describe. I'm happy to re-review once the metering is genuine. |
|
Yes, there is a bug in the token accounting inherited from the router
internals. Gonna fix it and resubmit. Thanks!
…On Sat, 29 Aug 2026 at 23:45, Yifan Lu ***@***.***> wrote:
*yl231* left a comment (RouteWorks/RouterArena#191)
<#191 (comment)>
Thanks for the submission, @sashakolpakov
<https://github.com/sashakolpakov>. Maintainer review — I can't accept
the current result because the *cost is not real*, and the cost axis is
what places this at the top of the board.
Auditing r2-reasonometry-router.json:
- 95% of queries (8,012 / 8,400) route to THUDM/GLM-4-9B-0414 on a
local-ollama endpoint, emitting *bare multiple-choice letters* (e.g.
"\nC", 3 output tokens).
- The decisive issue: *input_tokens is a constant 40 on every row*,
regardless of prompt length — and the prompts vary from tens to hundreds of
tokens. Real tokenizers never return an identical input count across
different prompts. (input, output, total) = (40, 3, 43) appears on
*69.7%* of all queries.
Hard-coded token counts produce the reported *$0.04/1K*, which is what
yields the #1 <#1> Arena
score — so the ranking is an artifact of the fabricated metering, not the
router. This also contradicts the PR's own description of a "multi-round /
API-calling reasoning router."
To be reconsidered, please resubmit with *real token accounting* from the
actual model calls (true per-prompt input tokens and full completion
output), and predictions that reflect the multi-round behavior you
describe. I'm happy to re-review once the metering is genuine.
—
Reply to this email directly, view it on GitHub
<#191?email_source=notifications&email_token=ABIOQZKVTMQB2EC7IVWUAMT5MNFJPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBWGUYDMMRWGA2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5465062604>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABIOQZKOFE6SCFCBORQFSU35MNFJPAVCNFSNUABGKJSXA33TNF2G64TZHMYTANZQGM2DOMRTG45US43TOVSTWNJSGE2TSNBUGE4DTILWAI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
2 participants
The previous export path put internal projected token usage into the public
generated_result.token_usagefields instead of usage measured from the model calls. RouterArena then calculated its cost and Arena score from those invalid fields.