fix: raise max_tokens for AI analysis to avoid thinking-model truncation#241
Conversation
…el truncation Reasoning models (e.g. agnes-2.0-flash) enable thinking by default and emit reasoning_content. With the previous 1000/1200/800 token cap the thinking chain consumed the whole budget and hit finish_reason:"length" before the final JSON was written to content, causing repository analysis to fail. Bump the analysis and semantic-reranking token budgets to a shared ANALYSIS_MAX_TOKENS=4096 so the model can finish thinking and emit the answer. max_tokens is a standard OpenAI param accepted by any compatible endpoint.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthrough
ChangesAI analysis token budgeting
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request replaces hardcoded token limits across different AI service calls with a single static constant, ANALYSIS_MAX_TOKENS. The reviewer suggests defining separate constants for repository analysis and search reranking (e.g., RERANKING_MAX_TOKENS) to allow independent tuning of token limits, preventing unintended latency and cost increases.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| private config: AIConfig; | ||
| private language: string; | ||
| private static readonly ANALYSIS_MAX_ATTEMPTS = 3; | ||
| private static readonly ANALYSIS_MAX_TOKENS = 4096; |
There was a problem hiding this comment.
Sharing a single constant ANALYSIS_MAX_TOKENS for both repository analysis and search reranking reduces maintainability and flexibility. Repository analysis (which generates a detailed summary, tags, and platforms) and search reranking (which only outputs a list of IDs) have very different output sizes and latency requirements.
If you want to adjust the token limit for repository analysis in the future (e.g., increasing it to 8192), it will unintentionally increase the limit for search reranking as well, potentially leading to higher latency and costs for search queries.
Consider defining separate constants for these tasks so they can be tuned independently.
| private static readonly ANALYSIS_MAX_TOKENS = 4096; | |
| private static readonly ANALYSIS_MAX_TOKENS = 4096; | |
| private static readonly RERANKING_MAX_TOKENS = 4096; |
| user: `Query: ${query}\n\nGists:\n${this.sanitizeForPrompt(gistSummaries)}`, | ||
| temperature: 0.1, | ||
| maxTokens: 1200, | ||
| maxTokens: AIService.ANALYSIS_MAX_TOKENS, |
There was a problem hiding this comment.
Use the separate RERANKING_MAX_TOKENS constant here to allow independent tuning of the token limit for Gist search reranking.
| maxTokens: AIService.ANALYSIS_MAX_TOKENS, | |
| maxTokens: AIService.RERANKING_MAX_TOKENS, |
| user: `Query: ${query}\n\nRepositories:\n${this.sanitizeForPrompt(repoSummaries)}`, | ||
| temperature: 0.1, | ||
| maxTokens: 800, | ||
| maxTokens: AIService.ANALYSIS_MAX_TOKENS, |
There was a problem hiding this comment.
Use the separate RERANKING_MAX_TOKENS constant here to allow independent tuning of the token limit for repository search reranking.
| maxTokens: AIService.ANALYSIS_MAX_TOKENS, | |
| maxTokens: AIService.RERANKING_MAX_TOKENS, |
Address gemini-code-assist review: repository analysis and search reranking have different output sizes/latency needs, so decouple their token limits into ANALYSIS_MAX_TOKENS and RERANKING_MAX_TOKENS for independent tuning.
1 participant
Summary
agnes-2.0-flash) enable thinking by default and emitreasoning_content. With the previous 1000/1200/800 token caps, the thinking chain consumed the entire budget and hitfinish_reason:"length"before the final JSON was written tocontent, causing repository AI analysis to fail (empty content → "No content received from AI service").ANALYSIS_MAX_TOKENS = 4096so the model can finish thinking and emit the answer.Why this approach
max_tokensis a standard OpenAI-compatible parameter accepted by any endpoint — no risk of 400s from unknown/vendor-specific fields, and no model/provider database to maintain.chat_template_kwargs/enable_thinking/thinking.typethat vary per provider and can error on strict relays). It simply gives the token budget enough room for thinking + final JSON.Changes
src/services/aiService.ts:private static readonly ANALYSIS_MAX_TOKENS = 4096;analyzeRepository:maxTokens1000 → 4096searchGistsWithReranking:maxTokens1200 → 4096searchRepositoriesWithSemanticReranking:maxTokens800 → 4096Test plan
finish_reason:"stop",contentcontains valid JSON, and analysis succeeds.tsc --noEmitandeslintpass (verified locally).🤖 Generated with OpenCode
Summary by CodeRabbit