feat: vision describe intercept, effort mapping, nonessential-traffic shutoff - #11
Merged
Andres Thor Saemundsson (andresthor) merged 9 commits intoSep 15, 2026
Conversation
- capability-driven: the intercept is a no-op when the resolved model reports image_input:true, so a future multimodal corti-s1 passes images through with no code change - describe sidecall returns a JSON object (response_format: json_object, Corti-honored) with typed fields: verbatim text_content, description, answer, uncertainties - defaults to the instant sonnet tier (corti-s1-mini-instant): multimodal and non-reasoning, so no thinking cap and low latency - context-aware: the user's question is passed to the vision model so the answer field targets it - descriptions cached per image (image hash only) and byte-identical across turns for A1 prefix-cache stability
- export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC so bootstrap, registry, telemetry, crash reports, plugins, Projects, /bug, /feedback stop bypassing the gateway
…ning levels - the picker sent output_config.effort (six levels) but the proxy dropped it and forced reasoning_effort=medium via adaptive thinking, so the effort axis was a no-op - map low/medium/high→high and xhigh/max/ultracode→max; Corti only supports high and max - adaptive thinking enables reasoning without pinning depth; output_config.effort wins over the legacy budget mapping, advisorEffort wins over both - gate reasoning_effort on a thinking block so non-reasoning requests get no effort level
- kept despite being inert: the catalog-driven derivation is correct and would take effect under a provider config (Bedrock/Vertex/Foundry/Mantle); the picker and send-time effort are shaped in translate.mjs instead
- strip system-reminders: a 3656-char CLAUDE.md replay displaced the real 117-char question - drop answer: an image-hash cache key replays a request-dependent field stale on every later turn - add palette: prose colour naming rendered a #141413 ground as "black" - pin temperature: unpinned decoding made prompt changes unmeasurable
- a re-Read of a pasted image carried a fresh tool_use_id and re-described identical bytes - three reads of one image gave three palettes, and the model trusted the last
- replaces the don't-pad line, which fought an existing terseness prior - image_type dropped: it varied across calls on one image and nothing routed on it - docs carry the cache-key fix from the previous commit, edited in the same paragraphs
Andres Thor Saemundsson (andresthor)
marked this pull request as ready for review
September 14, 2026 16:53
Copilot started reviewing on behalf of
Andres Thor Saemundsson (andresthor)
September 14, 2026 16:54
View session
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved gateway reliability, routing, caching, and effort-handling issues remain.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds capability-driven image descriptions, effort mapping, and suppression of nonessential Claude Code traffic.
Changes:
- Adds cached vision sidecalls and image fallbacks.
- Maps effort levels to supported Corti values.
- Updates tests, documentation, and wrapper configuration.
File summaries
| File | Summary |
|---|---|
translate.mjs |
Image interception, caching, and effort mapping |
test/translate.sh |
Translation and image-interception tests |
lib/vision-describe-prompt.txt |
Vision description prompt |
lib/models.mjs |
Model capability documentation |
GUIDE.md |
User-facing documentation |
gateway.mjs |
Capability discovery and vision sidecalls |
CLAUDE.md |
Architecture and invariant documentation |
bin/corti-bridge |
Nonessential traffic shutdown |
Review details
Suppressed comments (4)
translate.mjs:1263
- When
body.thinkingis an object with a non-reasoning type (for example,{ type: "disabled" }), this branch still assignsreasoning_effort: "high"because it never checksth.type. That violates the non-reasoning guard and can make an explicitly disabled request fail upstream with the same 400 this change is meant to avoid. Restrict this assignment to theenabled/adaptivetypes, leaving the advisor override outside this branch.
if (th && typeof th === "object") {
translate.mjs:1277
mapEffortestablishes that Corti only acceptshighandmax, but this advisor override bypasses it. The documentedCORTI_ADVISOR_EFFORT=medium(andlow) therefore sends an unsupported value upstream and makes advisor continuations fail with 400; normalize the override through the same mapping.
if (opts?.advisorEffort) req.reasoning_effort = opts.advisorEffort;
translate.mjs:754
- The cache is populated only after
describeImageresolves, so two concurrent requests for the same session and image can both miss and run nondeterministic vision calls, then send different descriptions in the same conversation. That violates the byte-stability invariant this cache is intended to provide; deduplicate in-flight descriptions by session and image key or serialize image interception per session.
let desc = key ? cache.get(key) : undefined;
if (desc === undefined) {
const got = await describeImage(img, context);
translate.mjs:711
- This context extractor ignores string-form message content. A normal user turn such as
content: "what is wrong with this screenshot?"is skipped, so an image in a later tool result receives no request context and the vision sidecall cannot focus on the user's question. Normalize string content to a text block before filtering.
if (!msg || !Array.isArray(msg.content)) continue;
const text = msg.content
- Files reviewed: 7/8 changed files
- Comments generated: 5
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- add 30s timeout and non-200 status check to the /models capability fetch so a hung or errored upstream self-heals instead of parking imageModelsLoading forever - hoist mapEffort to module scope, raise image cache cap to 128, drop the off-vocabulary medium advisor-effort recommendation from GUIDE and the gateway comment - document the nonessential-traffic shutoff as a known degradation with its escape hatch
…ets the haiku tier - extract load_models_env, called from the default path and --restart, so visionModel()'s ANTHROPIC_DEFAULT_HAIKU_MODEL fallback is no longer dead in the gateway process - stamp CORTI_VISION_MODEL from the haiku tier (explicit override still wins) and document the restart requirement
Maksim Filipenka (mfilipenka)
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three fixes for behaviors the harness exercised but the gateway dropped or crashed on.
Vision describe for blind models
corti-s1(opus) is blind: aReadon an image returned a block Corti rejected with400 "not a multimodal model", killing the session. Now each image is described by a sighted sidecall and replaced with text, so the blind primary never sees one. Capability-driven — a future multimodalcorti-s1passes images through with no code change.How an image-bearing request is handled:
tool_use_id— a re-Read carries a fresh id; keying on it re-described identical bytes into three different palettes and the model trusted the last.answerfield was tried and removed (it replayed stale and made the vision model reply as the caller).corti-s1-mini-instant(multimodal, non-reasoning, low latency);CORTI_VISION_MODELoverrides.Effort mapping
The picker sent
output_config.effort(six levels) but the proxy dropped it and forcedreasoning_effort=medium— the axis was a no-op. Now collapsed onto Corti's two real levels:low/medium/high → high,xhigh/max/ultracode → max.output_config.effortwins over the legacy budget mapping;reasoning_effortis gated on athinkingblock (a non-reasoning request 400s on it).Nonessential traffic
bin/corti-bridgeexportsCLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1— bootstrap, registry, telemetry, crash reports, plugins, Projects,/bug,/feedbackall bypassedANTHROPIC_BASE_URLbefore.