Shunt submission hand-off — RouterArena (sub_10) - #205
Open
KookaS wants to merge 3 commits into
Open
Conversation
Author
|
/evaluate |
…ion reproducible + CI-green - router module: SHUNT_PIN constant + seed-dir-first resolution (router_inference/router/shunt_seed/ preferred, env/artifact root fallback), typed _PRIOR (fixes mypy Collection[str] index errors on 3.9-3.12), no unused _import_shunt assignment (ruff F841); import registered in __all__ (ruff F401). - shunt_bench_prior: _embed_many callable narrowing (mypy 'None' not callable), drop zip(strict=) for py3.9 mypy, SPDX header, no codespell hits. - bundle router_inference/router/shunt_seed/ (sample + embeddings verbatim, results filtered to the menu models) + REPRODUCE.md + .codespellrc so a clean checkout at the pin regenerates byte-identical predictions. - predictions files untouched.
…nch_prior) Prediction choices are now all deepseek/deepseek-v4-flash, produced by Shunt's real session_cascade engine at the pinned commit; answers reused from the recorded cache (no re-inference).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Router shunt (Shunt, @KookaS, https://github.com/KookaS/shunt)
routes every prompt over the menu deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro through Shunt's real
knn_semantic_cascadeengine — the shipped seeded kNN pick over an outcome index built from the prior-seeded verified-outcome corpus (a labeled QA-domain corpus: MMLU + GSM8K prompts, deterministic graders, no LLM judge, measured at real provider cost for the menu models — RouterArena's own generic-QA domain). Choices are corpus-driven: a prompt whose nearest cells lack evidence the cheap rung (deepseek/deepseek-v4-flash) succeeds is escalated to the rung the corpus evidence supports. RouterArena's stateless single-turn regime gives Shunt no verified-outcome channel, so the router's signal is the measured semantic neighbourhood itself. Thedecision files are reproducible from this worktree at pinned upstream cff9659.
model_inference.pyprovider mapping rewritten (or added) to that provider for the run and reverted after; the change is LOCAL ONLY and must not be committed unless you want upstream to reproduce native-DeepSeek inference without an OPENROUTER key.