fix(metrics): align WandB axes and weighted minibatch stats#410
Closed
Hecate0821 wants to merge 1 commit into
Closed
fix(metrics): align WandB axes and weighted minibatch stats#410Hecate0821 wants to merge 1 commit into
Hecate0821 wants to merge 1 commit into
Conversation
VigSri
pushed a commit
to VigSri/cookbook
that referenced
this pull request
May 5, 2026
…w-ai#405) * feat(verifier): empirical renderer probe (Phase 0) Adds training/verifier/, the first slice of the renderer verifier: an empirical probe that pairs each rendered token's renderer claim (chunk source, training weight) against deployment-observed provenance (was this token actually emitted by the model, or was it template-injected). The probe is observational only — verdicts come in follow-up PRs (L1 spec-driven CPU checks, L2 corpus-scale checks) and consume the same JSON envelope. What's in this PR - training/verifier/probe.py: render → call deployed model → render full transcript → align tokens → emit JSON artifact with audit table (chunk_source, renderer_claim_weight, provenance per token). - training/verifier/cli.py + __main__.py: ``python -m training.verifier render --renderer glm5 --tokenizer-model ... --model ... --input msgs.json --output probe.json``. - training/verifier/README.md: artifact schema and authoring workflow (what spec authors look for in the audit table). - training/verifier/examples/: two GLM5 input fixtures. - training/tests/unit/test_verifier_probe.py: network-free smoke test with a stub Fireworks-shaped client and a toy renderer; asserts artifact shape, sanity flags, and the provenance partition (prompt_hard_append / native_generated / trailing_hard_append / tokenization_diverged). Why The bug class behind GLM5 fw-ai#400 (a trailing role-tag with weight=1 the model never emits) is exactly the disagreement the audit table surfaces row-by-row. The probe is the authoring tool for the YAML chunk_rules spec the L1 verifier will assert against; without it, spec authoring relies on imagining the token stream from raw Jinja, which is how the original bug shipped. Out of scope - No spec format, no L1/L2 verdict layers (follow-up PRs). - No deployment lifecycle: ``--model`` takes an existing Fireworks model identifier; pair with the existing rl_loop-style spinup helper. - No multimodal / VL renderer support; v1 is text-only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: add minimal probe deployment spinup helper Adds training/verifier/spinup_deployment.py — a small CLI wrapper around DeploymentManager + DeploymentConfig (the same path training/recipes/rl_loop.py uses) so the probe has a documented way to provision a serving deployment. Shape resolution mirrors the rest of the cookbook: * deploymentShapes/<id>/versions/<v> → used as-is * deploymentShapes/<id> → latest validated version * trainingShapes/<id>/versions/<v> → that version's pinned deployment_shape_version * trainingShapes/<id> → latest validated training- shape version's pinned deployment_shape_version Default --shape is the GLM5-on-B300 deployment shape (accounts/fireworks/deploymentShapes/glm-5p1-b300/versions/jqami1br). README updated with the end-to-end flow: MODEL=$(python -m training.verifier.spinup_deployment up ... | tail -1) python -m training.verifier render --model "$MODEL" ... python -m training.verifier.spinup_deployment down --deployment-id ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: fix probe round-trip — strip echoed prompt, parse via renderer Two corrections found while running the probe end-to-end against serverless GLM5 (accounts/fireworks/models/glm-5p1): 1. Echo handling. With ``echo=True`` the Fireworks gateway returns ``completion_token_ids`` and ``message.content`` as prompt+completion concatenated. The first iteration treated them as completion-only, so every assistant-turn token was classified ``tokenization_diverged``. The probe now detects when ``completion_token_ids`` starts with ``prompt_token_ids`` and slices the prompt prefix off (recording ``echo_prompt_stripped`` in ``sanity``). The completion text is re-decoded from the sliced ids so it doesn't carry the echoed prompt. 2. Round-trip via parse_response. Camp A renderers (GLM5) emit the trailing role tag as ``stop_overlap``. When the model itself emits the same tag (which it does — the model is trained to use it as a stop signal), feeding the raw completion text back as ``content`` doubles up: once embedded in the content string, once as the renderer's stop_overlap. The probe now hands the completion tokens to ``renderer.parse_response`` first, then re-renders the structured ``Message`` it returns. This is the renderer's own definition of "what the assistant turn was", so the round-trip stays non-circular while the alignment becomes 1:1 with the API's emission. Falls back to raw-content feed when parse_response signals failure (``parse_response_ok`` recorded in sanity). Other small changes: * Default ``--max-tokens`` 256 → 1024. Thinking-enabled models like GLM5 routinely emit 100+ tokens before a natural stop; under the old default the probe hit ``stop_reason: length`` and the trailing-token bucket couldn't be empirically classified. * New ``glm5_short_answer.json`` example fixture — a math question with a one-integer constraint, designed so the model finishes naturally inside 1024 tokens for a clean stop_reason='stop' read. * Sanity additions: ``echo_prompt_stripped``, ``parse_response_ok``, ``completion_token_count``, ``completion_stop_reason``. * Test rework: the toy tokenizer now uses greedy-longest-match so decode→encode round-trips cleanly, and the second test was rewritten to cover the new echo-stripping path (the prior "tokenization_diverged" assertion no longer applies once parse_response is in the loop). E2E validation against serverless GLM5: artifact lands clean (provenance partition: 27 prompt_hard_append + 127 native_generated, zero divergence, zero spurious trailing rows). Confirms that post-fw-ai#400 GLM5 stop_overlap weight=1.0 is empirically validated — the model itself emits the trailing <|user|> as its Camp A stop signal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: trust renderer.build_supervised_example for weights; add inspect Two fixes / additions, both surfaced by reading the GLM5 audit table from the previous run: 1. Renderer's claim_weight is now read from ``renderer.build_supervised_example(messages, train_on_what)``'s weights tensor, not re-derived from a base-class loop in the probe. The old approach was wrong for any renderer that customizes weighting — for GLM5 specifically it reported the leading ``<think>`` token as weight=1.0, when GLM5's actual override splits that token off and assigns it weight=0 (per the renderer's docstring: "kept in rendered sequence for template parity but masked out of loss because it is already injected by the generation suffix"). This was also a circularity hazard: re-implementing weight logic in the verifier would have made the verifier validate the renderer against the verifier's idea of what the renderer should do, which catches nothing. New design: ``_structural_spans`` walks render_message for chunk_source / role / msg_idx attribution only — no weight computation. ``build_supervised_example`` is the single source of truth for weights. The two walks must agree on token identity; if they don't (renderer customization that changes tokens, not just chunk boundaries), ``sanity.structural_walk_token_match = false`` flags it. 2. ``python -m training.verifier inspect <probe.json>`` — pretty-prints sanity, provenance counts, and a digest of the audit table (chunk boundaries, every special-source row, every empirically suspect row). Drops the need for ad-hoc ``python -c`` snippets when reviewing a probe artifact. ``--all`` and ``--filter <provenance>`` flags for narrower views. E2E re-validation against serverless GLM5 with the short_answer fixture: idx=26 (the leading ``<think>``) now reads ``w=0.0 prov=prompt_hard_append`` — both halves of the verification principle agree, GLM5 post-fw-ai#400 stays clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: unit-test inspect renderer Adds test_inspect_renders_structured_summary covering the sanity / provenance / audit-table sections and the --filter narrowing path. Keeps the new CLI surface in CI without needing network or HF model downloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: cross-renderer e2e validation (Qwen3 Camp B) Probed accounts/fireworks/models/qwen3-8b with the qwen3 renderer. Result: 240 audit rows, 0 tokenization_diverged, the trailing <|im_end|> is correctly classified output / w=1.0 / native_generated (Camp B layout — no stop_overlap row, by design). Confirms the probe is renderer-agnostic across the two structural shapes (Camp A's next-role-tag-as-stop and Camp B's inline end marker) without code changes. Adds qwen3_short_answer.json fixture and a "Verified against" table in the README so the next renderer addition (Gemma4, MiniMax, Nemotron, Kimi K2.5) has a concrete shape comparison to slot into. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: add sweep harness over serverless-reachable renderers Adds python -m training.verifier.sweep that runs the probe against a fixed catalogue of (renderer, model, input) rows wired to PR-history bug refs, and emits both per-row JSON artifacts and a summary.json plus a compact stdout table. The sweep is a regression harness (records and reports), not a test runner — failures stay observable without forcing a CI red. Initial run against accessible Fireworks serverless models: glm5-single-turn PASS cookbook#384,fw-ai#389,fw-ai#400 glm5-multi-turn PASS cookbook#397,fw-ai#400 qwen3-thinking-single-turn PASS tinker#178,fw-ai#247,fw-ai#341 qwen3-disable-thinking-single-turn FAIL: HF prompt parity deepseekv3-single-turn FAIL: HF prompt parity deepseekv3-thinking-single-turn FAIL: HF prompt parity The three FAILs are real prompt-parity disagreements between the cookbook renderer and the Fireworks serverless serving template: * qwen3_disable_thinking renders a 4-token suffix ``<think>\\n\\n</think>\\n\\n`` to force the model to skip thinking; serverless qwen3-8b just emits ``<think>\\n``. * deepseekv3 / deepseekv3_thinking prefill ``</think>`` / ``<think>`` after the assistant role tag (per upstream tinker fw-ai#267); the serverless template stops at ``<|Assistant|>`` without the prefill. These are exactly the kind of cross-stack disagreement the probe is designed to surface. The probe doesn't decide which side is correct — the human does, by reading the audit table and the model card — but it records the disagreement faithfully so it can no longer go unnoticed. UNREACHABLE_BUGS in the sweep module documents bug classes not exercised here (Nemotron / MiniMax / Gemma4 — not on serverless; multimodal — out of scope; per-message weight=0 — pre-renderer normalize_messages bug). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: expand sweep to all serverless-callable Fireworks models Catalogue grew from 6 to 14 (renderer, model, input) rows covering every chat model we can hit serverless from a stock dev API key: PASS rows (renderer behaviour confirmed against serving template): glm5 x glm-5p1 single + multi-turn cookbook#384,fw-ai#389,fw-ai#397,fw-ai#400 glm5 x glm-5 base cookbook#384 qwen3 (think) x qwen3-8b single + multi-turn tinker#142,fw-ai#178,fw-ai#247,fw-ai#341 kimi_k25 x kimi-k2p5 tinker#248,fw-ai#384,fw-ai#393,fw-ai#410 kimi_k25 x kimi-k2p6 (cookbook#357 routing) cookbook#357,tinker#410 FAIL: HF prompt parity (probe surfaces real disagreement at the renderer-vs-serving-template seam): qwen3_disable_thinking x qwen3-8b kimi_k25_disable_thinking x kimi-k2p5 deepseekv3 x deepseek-v3p1 deepseekv3_thinking x deepseek-v3p1 minimax_m2 x minimax-m2p7 llama3 x llama-v3p3-70b single + multi-turn The disable-thinking variants emit a suffix the serving template doesn't (renderer forces think-skip, gateway defaults to thinking enabled). DeepSeek renderers prefill <think> per upstream tinker#267; the serving template hasn't followed. minimax_m2 has an off-by-one extra newline after <think> in the generation suffix. Llama 3.3 needs a closer look — the API may be returning prompt_token_ids that include the completion when echo=True, so that row's "FAIL" is partly a probe-extraction artefact rather than a pure renderer bug. Other catalogue notes (in code comments): * qwen3 4b/32b/30b-a3b/instruct-2507 variants and qwen3p5-27b are in the inference catalogue listing but return 404 on chat.completions; not actually serverless from this dev API key. * deepseek-v4-pro is callable but doesn't surface prompt_token_ids/token_ids, so the probe's echo-based parity check has no signal there. * pyroworks-qwen3-1p7b is "no healthy upstream" (scaled to zero). Verdict heuristic now reports stop_reason=length BEFORE parse_response_ok=False — a length-truncated completion legitimately fails the parser, so flagging that as a parse bug was a false positive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: add CPU HF chat-template parity layer (CI gate) Adds the second of the two verifier probes — CPU-only renderer↔HF parity, separate from the live empirical probe so PR CI doesn't depend on a deployment and the live-only checks don't have to pretend to validate upstream intent. Layer split (now in README): CPU — does the renderer match upstream HF's canonical chat template byte-for-byte? hf_parity.py + tests/unit/test_renderer_hf_parity.py pytest assertions, no JSON artifact, runs on every PR. Live — does the renderer match what the deployed model actually emits? probe.py (verifier render CLI) → JSON audit-table artifact. Manual / nightly, needs a Fireworks API key. A renderer can pass one and still fail the other; both are needed. Combining the two disambiguates the renderer-vs-gateway question: * CPU PASS + live FAIL → gateway is stale * CPU FAIL + live FAIL → renderer is wrong * CPU PASS + live PASS → fully validated CPU layer initial cases (all PASS except the one xfail): glm5 single + multi-turn PASS qwen3 (default thinking) single + multi PASS qwen3_disable_thinking PASS ← resolves the live sweep FAIL: gateway is stale on the enable_thinking=False flag, NOT a renderer bug. kimi_k25 single PASS minimax_m2 single XFAIL ← confirms the live sweep FAIL is a real renderer bug (extra \\n after <think> in generation suffix). The xfail flips to PASS automatically when the renderer is fixed; a new regression in the opposite direction goes red. Tests skip cleanly when the upstream tokenizer can't be downloaded (gated repo, no network) so CI without HF Hub access still completes. CI integration: cookbook's pyproject already has testpaths=["training/tests"], so the new test file is picked up without additional configuration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: add React-based viewer for live probe artifacts Single-file React app at training/verifier/viewer/index.html that renders a probe JSON artifact as a colour-coded token stream. Loaded via React 18 + Babel from CDN, no build step. Open the file directly in a browser, pick (or drop) a probe JSON, see: * Background colour = provenance bucket (prompt_hard_append / native_generated / trailing_hard_append / tokenization_diverged). * Left border colour = chunk_source (bos / header / output / stop_overlap / generation_suffix). * Underline = renderer_claim_weight > 0 (trainable token). * Magenta + bold = token id is in tokenizer.special_tokens (the map from tokenizer.json's special-tokens declaration; the artifact already carries this). * Hover = full audit row in tooltip. Filters at the top let the reader hide specific provenance buckets, restrict to trainable tokens only, or restrict to special tokens only. Also surfaces metadata (renderer, tokenizer, sampling), sanity flags as ok/fail pills, and provenance / chunk_source counts. Visualization is *only* for the live empirical probe — the CPU HF parity layer is assertion-style (test framework is the GUI). README updated to document the layer split. The viewer is CDN-only and dependency-free so it can ship inside the repo and run from any developer's filesystem without a Node toolchain. If we later want a richer GUI (multi-artifact diff, sweep summary view, or a hosted webapp), the JSON envelope is the long-lived contract — the viewer is just a lens on top of it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: clean up dispatch — serverless default + --deployment-id auto-resolve The design now has exactly two parts: 1. ``training/verifier/spinup_deployment.py`` — owns deployment lifecycle (up / wait-ready / down). Unchanged in this commit; only documentation changes around how it pairs with fw-ai#2. 2. ``training/verifier/render`` — owns probing. Picks where to send the request based on the flags it gets: --deployment-id <id> → accounts/<account>/deployments/<id>, with <account> resolved automatically from the API key via DeploymentManager. Mutually exclusive with --model. --model <full-id> → exactly that string (mode=explicit). neither → renderer's registered Fireworks serverless default (mode=serverless). neither + no default → error pointed at spinup_deployment. The artifact's ``deployment`` block now records ``mode`` plus ``api_flags`` (echo / raw_output / return_token_ids) and ``extra_completion_kwargs`` so reviewers can see at a glance exactly what the gateway was asked to do. Per-renderer serverless defaults (RENDERER_SERVERLESS_DEFAULTS in probe.py) cover the renderers we verified callable in the sweep: glm5, qwen3 / qwen3_disable_thinking, kimi_k25 / kimi_k25_disable_thinking, deepseekv3 + thinking variants, minimax_m2, llama3. Renderers without a registered default (gemma4, nemotron3) trigger an explicit error message that points at spinup_deployment. React viewer: * Metadata splits into a "Renderer args" panel (name, train_on_what, every config.* key) and a "Deployment / API args" panel (mode pill, model, deployment_id, sampling.{temperature, max_tokens}, every api_flags.{echo, raw_output, return_token_ids} with ok/fail pills, extra_completion_kwargs, tool count). * The user-visible knobs (disable_thinking via renderer.config, echo / raw_output via api_flags, etc.) now have their own labelled rows instead of being buried in a JSON dump. README: * New table showing the four dispatch cases. * Default flow shrunk to "no --model, no --deployment-id" (just the serverless default, the simplest possible invocation). * Personal-deployment flow shows --deployment-id auto-resolve so no shell pipe through ``tail -1`` is needed. E2E spot-check against serverless GLM5 (no --model passed): dispatch=serverless model=accounts/fireworks/models/glm-5p1 api_flags={'echo':True,'raw_output':True,'return_token_ids':True} sampling={'temperature':0.0,'max_tokens':1024} Tests: 9 pass + 1 xfail (unchanged). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: live GUI — dev server + interactive form in the React viewer Adds a stdlib HTTP server (training/verifier/serve.py) that hosts the React viewer and exposes a single POST /probe endpoint. The viewer gains a "Live probe" form with every user-visible knob: * renderer (datalist of registered names; tokenizer auto-fills) * tokenizer_model (HF id; editable) * model | deployment_id (mutually exclusive; blank both → serverless default; live "dispatch preview" line shows what will be used) * max_tokens, temperature, train_on_what * tools (optional JSON array) * messages (system/user/assistant/tool rows, add/remove inline) Submitting POSTs to /probe → server runs run_probe with the form data → returns the same artifact JSON the CLI writes → the existing audit table view renders it in place. The probe-fixed API flags (echo / raw_output / return_token_ids) are spelled out next to the form so the user can see what's actually sent to the gateway, and they're already surfaced in the Deployment / API args panel below. Refactor: * probe.py grows ``resolve_dispatch`` and ``DispatchError``; cli.py's _resolve_dispatch is now a thin argparse adapter so CLI and server share the same precedence rules (--deployment-id auto- resolve → --model → renderer's serverless default → error). * Tokenizer + Fireworks client are cached at module level in serve.py so repeated browser submissions don't repay HF download / SDK auth on every keystroke. Verified end-to-end: curl /health → {"ok": true} curl -X POST /probe (glm5, 2+2 prompt) → mode=serverless, model=accounts/fireworks/models/glm-5p1, 89 native tokens, 105 audit rows, stop_reason=stop. Existing JSON-load path is preserved (collapsed under a "Or load an existing probe JSON" disclosure). Tests: 9 pass + 1 xfail. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: batch-native UI + YAML-driven inspection rules + triage workflow UI / batch model - Each "Run probe" or file-load appends a case; cases stack with per-case delete + Clear-all button. Counts panel removed. - Per-case sanity flags / renderer args / deployment args (collapsed by default, after the token stream). - Token stream + sticky right-side detail sidebar; amber-rippling background marks tokens that match an inspection rule. - Unified attribute filter chips (provenance / chunk_source / trainable / special / inspect-flag), shared across cases. Inspection rules — single source of truth - New training/verifier/inspect_rules.yaml describes "worth inspecting" attribute combinations (id / when / reason). Generic equality matcher in inspect_rules.py; viewer fetches /inspect_rules and evaluates the same rules client-side. Triage + reachability - training/verifier/triage.py loops a JSON corpus through run_probe, pre-flights with a serverless ping, and writes a session file consumed by serve.py --session-file (auto-seeds the GUI on mount). - training/verifier/check_renderers.py sweeps every registered renderer's serverless model and reports reachability. - training/verifier/default_prompts.json ships an 8-case starter corpus. Skill - training/verifier/SKILL.md documents the workflow end-to-end: rules → interactive vs batch → renderer/serverless preflight → GUI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: split into utils/ + rules/, drop sweep + check_renderers Folder layout - utils/ — engine modules (probe, inspect_rules, inspect, hf_parity) - rules/ — data files (inspect_rules.yaml, default_prompts.json) - top level — CLI entry points only (cli, serve, triage, spinup_deployment) - SKILL.md drives the test flow; the package exposes utilities, not many overlapping pre-packaged scripts. Removals - sweep.py: parallel verdict heuristic, not the inspect_rules.yaml source of truth — drop. - check_renderers.py: redundant with triage's pre-flight ping; SKILL.md describes how to verify reachability with a one-prompt triage run. User-facing CLI paths unchanged - python -m training.verifier render | inspect - python -m training.verifier.serve - python -m training.verifier.triage - python -m training.verifier.spinup_deployment Tests updated; 9 passed + 1 xfail (existing MiniMax xfail). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: SKILL.md is the only doc; drop default prompt catalogue - Delete README.md — SKILL.md is the canonical entry point. - Delete rules/default_prompts.json — a baked-in corpus is itself a pre-packaged workflow. The package now exposes utilities and a thin triage runner; users author their own prompt JSON for the cases that exercise the surface they care about. - triage.py: --prompts is now required (no default fallback). - triage.sh + SKILL.md: updated to require an explicit prompts.json arg. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: surface the renderer verifier in cookbook README + dev skill - README.md: add `verifier/` to the repository tree and a short "Verifying a renderer" section pointing at training/verifier/SKILL.md. - skills/dev/SKILL.md: two new task-routing rows so agents who hit a renderer-shaped problem land on the verifier doc directly. - skills/dev/references/tools.md: header note that the verifier is a separate standalone tool with its own SKILL.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * viewer: renderer field is a real <select>, not a datalist autocomplete The previous <input list="renderers"> only surfaces suggestions when the user clicks into the field — on most browsers it looks like a plain text box, so users couldn't tell other renderers were even available and only ran glm5. A plain <select> exposes the full RENDERER_TOKENIZER_DEFAULTS list at a glance. Tokenizer_model stays as a free-text input so users can point at any HuggingFace tokenizer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: drop static renderer→model defaults; query Fireworks live The cookbook used to ship RENDERER_SERVERLESS_DEFAULTS — a hard-coded dict of Fireworks model ids per renderer. It went stale every time Fireworks bumped a serverless model. SKILL.md owns the human-readable mapping; the runtime queries the live API for what's actually available. probe.py - Remove RENDERER_SERVERLESS_DEFAULTS and serverless_default_for(). - resolve_dispatch now requires --model or --deployment-id; "serverless" mode is gone. The error message points at the /models endpoint and Fireworks().models.list() so the caller can discover what to pass. serve.py - /registered → drop. Replaced with two live endpoints: * /renderers — calls tinker_cookbook.get_registered_renderer_names() so the GUI dropdown reflects what's actually registered. * /models — calls Fireworks().models.list(account_id="fireworks") so the model dropdown reflects the live Fireworks catalogue. viewer/index.html - Remove RENDERER_TOKENIZER_DEFAULTS — no static mappings on the client. - Renderer field becomes a dropdown populated from /renderers. - Model field becomes a dropdown populated from /models (with the /models error surfaced inline if it fails). - Drop the renderer→tokenizer auto-fill; tokenizer is a free-text input so users can pick any HF tokenizer. - Dispatch preview reads "ERROR: pick a model from the dropdown or enter a deployment_id" when neither is set, matching the new contract. triage.py - Pre-flight registration check uses is_renderer_registered() instead of dict membership. Aborts on NOT REGISTERED. SKILL.md - New section 1: "Pick a renderer / tokenizer / Fireworks model" with the discovery snippet and a starter reference table. - Renumber the rest; drop "serverless" from the dispatch-mode wording. Tests: 9 passed + 1 xfail (MiniMax xfail unchanged). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: /models filters to supports_serverless=true server-side The control plane's gateway Model proto has an OUTPUT_ONLY `supports_serverless` bool, and ListModels accepts an AIP-160 `filter` parameter (see fireworks/control_plane/protos/gateway/model.proto). Pass `filter="supports_serverless=true"` so the dev server only returns the models the GUI can actually probe via serverless — no client-side filtering, no walking the entire fireworks-account catalogue. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * viewer: top-of-page advisory — prefer RL deployments over serverless Serverless and RL deployments don't always render and tokenize the same chat: different runtimes, different optimizations, occasionally different chat templates. The page now surfaces this up front so users default to deployment_id when they have one, and treat serverless probes as approximations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * viewer: replace native file-picker button with a styled label The browser-default <input type="file"> button doesn't match anything else in the form. Hide it offscreen and trigger it from a styled label that uses the same border + radius + hover treatment as the other buttons. The selected file name renders next to it in mono. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * viewer: hard mutex on model vs deployment_id + serverless billing note - Frontend now enforces the mutual exclusion the backend already required: when `deployment_id` has a value the model dropdown is disabled, and vice versa. No more accidentally setting both and hitting the server-side reject. - A small amber inline note appears under the model dropdown when a serverless model is selected, reminding the user that those probes hit the live API and are billed at the model's standard per-token rate. The advisory at the top of the page already nudges users toward `deployment_id`; this surfaces the cost reminder right where the choice is being made. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * skills: split renderer impl + verifier into cookbook/skills/ - Move training/verifier/SKILL.md → skills/verifier/SKILL.md so it lives where the cookbook expects skills (next to skills/dev/). - New skills/renderer/SKILL.md — implementation guide for adding a new renderer: contract, training-mechanics invariants, common shape decisions with examples in the tree, dev loop, and gotchas. Pairs with the verifier skill. - training/verifier/README.md is now a minimal placeholder pointing at both skills, with a layout map and a placeholder for the GUI screenshot to be added next. - Cookbook README + dev skill cross-reference the new locations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: forward HF_TOKEN; wrap tokenizer-load errors with guidance The HF Hub error users hit when a tokenizer id is misspelled or the repo is gated is opaque. New utils/tokenizer.py:load_tokenizer(): - forwards $HF_TOKEN / $HUGGING_FACE_HUB_TOKEN through to AutoTokenizer.from_pretrained so gated repos work for authenticated users without code changes - catches OSError/ValueError from from_pretrained and re-raises with a RuntimeError that lists the three common causes (typo, gated repo, no network) and mentions the ~/.cache/huggingface/ cache so users know subsequent loads are offline-friendly serve._tokenizer, triage._build_tokenizer, cli._build_tokenizer all funnel through the new helper. skills/verifier/SKILL.md mentions HF_TOKEN as an optional prereq and explains the cache behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: full renderer catalog with tokenizer auto-fill Why only 4 renderers showed before: get_registered_renderer_names() returns custom-registered renderers only. tinker_cookbook's ~20 built-in renderers live in get_renderer()'s elif ladder, never registered. serve.py now ships a static RENDERER_TOKENIZER_DEFAULTS catalog covering: - tinker_cookbook built-ins: role_colon, llama3, qwen3 (+ vl, instruct, disable-thinking variants), qwen3_5, deepseekv3 (+ thinking), kimi_k2, kimi_k25, nemotron3, gpt_oss (no_sys / low / medium / high reasoning) - cookbook-local: gemma4, glm5, minimax_m2, nemotron - anticipated next: kimi_k26, kimi_k26_disable_thinking, qwen3_6, qwen3_6_disable_thinking — surface them in the dropdown so they show up; selecting one before its renderer module exists yields a clean "renderer not registered" error. Also force-import training.renderer at module load so cookbook renderers are in the registry when /renderers fires. GUI: tokenizer auto-fills from the catalog when you pick a renderer. The Fireworks Model proto has huggingface_files (uploaded blobs) but no canonical HF repo id, so static mapping is the cleanest answer — documented as the source of truth in skills/verifier/SKILL.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * viewer: fix blanked page — render renderer dropdown from {name, ...} The /renderers endpoint started returning objects when the catalog landed, but the dropdown was still doing <option value={r}>{r}</option> which throws "Objects are not valid as a React child" and blanks the entire page. Read .name explicitly and surface "not registered" inline on aspirational entries. Also auto-fill tokenizer_default on change (was only happening on initial mount). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * verifier: move under training/renderer/, embed UI screenshots in README The verifier validates renderers — colocating it under training/renderer/verifier/ makes that ownership obvious and matches how the cookbook organises related code (tests live next to what they test, etc). What moved: training/verifier/ → training/renderer/verifier/ What changed: - All Python imports retargeted to training.renderer.verifier.*. - Workspace-root scripts (run.sh, triage.sh, run-bug-*.sh) updated to invoke `python -m training.renderer.verifier.*`. - skills/verifier/SKILL.md, skills/renderer/SKILL.md, cookbook/README.md point at the new path. - training/renderer/verifier/README.md now embeds two screenshots (config form + token stream) from the new images/ subfolder so visitors see the GUI before reading the long-form skill. - Tests (test_verifier_probe.py, test_renderer_hf_parity.py) updated to the new module paths. CLI invocations are now `python -m training.renderer.verifier render` etc. — verbose, but it's the price of correct nesting. The shell scripts hide this; only direct CLI users will notice. 9 passed + 1 xfail. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closed because this split is no longer desired: the WandB metric-axis fix belongs in PR #382, and the weighted
fwd_bwdaggregation change is not needed.