docs: self-hosted model output-trimming issue + proposed fix

[Blackfrost-AI]

What

Adds docs/SELF-HOSTED-MODELS.md — a documented known-issue + proposed fix. No code changes; this is a design note so we can come back and implement the fix.

Why

When Context Cooler drives the Claude Code TUI as a harness for a self-hosted / small model, the model can read the cooler's compact summary as data loss and abandon ctx_execute, falling back to raw Bash. That loses both the token savings and the cooler's guardrails — observed live with a quantized local model (huihui-q8).

Root cause (cited in the doc)

• Trim defaults are hard-coded and aggressive: compactDefault cuts arrays to 5 / objects to scalars (src/lib/filter.ts:197,202-204); non-JSON text is cut to 5000 chars (src/tools/execute.ts:170).
• The response carries no signal that trimming happened or that the full output is indexed and retrievable via ctx_search. Frontier models infer this; smaller models don't.

Proposed fix (for a follow-up PR)

1. Env-configurable caps following the existing CTX_* convention (CTX_MAX_ARRAY_ITEMS, CTX_MAX_OBJECT_KEYS, CTX_MAX_TEXT_CHARS), defaults = current values → zero behavior change unless opted in.
2. A terse retrieval hint in the ctx_execute response when trimming occurs (truncated/retrieve_with) so any model knows the rest is recoverable.
3. Optional CTX_PROFILE=self-hosted to bump all three at once.

Interim mitigation

Harness-side only: the operating profile now tells the model a trimmed summary means "the rest is on disk, use ctx_search." That doesn't help operators who haven't tuned their profile — hence the code-side fix proposed here.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation
  • Added documentation describing a known issue affecting users running self-hosted models, detailing observed behavior with output trimming and proposed mitigations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2bd0785d-302e-49b5-832c-87f6c13c6453

📥 Commits

Reviewing files that changed from the base of the PR and between 333a81e and da22531.

📒 Files selected for processing (1)

• docs/SELF-HOSTED-MODELS.md

---

📝 Walkthrough

Walkthrough

Adds docs/SELF-HOSTED-MODELS.md, a 90-line document describing a known failure mode where local/quantized models stop using ctx_execute after trimmed output is misread as data loss. It records the root cause, an interim prompt-profile mitigation, and a proposed code fix using CTX_* env variables and a truncation hint in responses.

Changes

Self-hosted model trimming issue documentation

Layer / File(s)

Summary

Known issue, root cause, mitigation, and proposed fix docs/SELF-HOSTED-MODELS.md

Documents the ctx_execute trimming failure mode (scope and status), the observed fallback-to-Bash behavior, the two-part root cause (hard-coded trimming caps and missing truncation signal), an interim harness prompt-profile mitigation, the proposed CTX_* env-var configurable caps plus response-level truncation/retrieval hint, and acceptance criteria with references to filter.ts, execute.ts, and env.ts.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~4 minutes

Poem

🐇 A doc appeared in the warren one day,
Describing the trimming that led models astray.
"The output was cut and the cache hint was gone,"
So models fled Bash and just carried on.
With env vars and hints, the cooler stays strong—
The rabbit wrote notes so the fix won't take long! 🌿

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding documentation about a self-hosted model output-trimming issue and its proposed fix. It is concise, specific, and directly reflects the changeset content.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/self-hosted-output-trimming

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}