Skip to content

fix(go/plugins/googlegenai): do not resend the cached prefix inline - #6141

Open
a2105z wants to merge 2 commits into
genkit-ai:mainfrom
a2105z:docs/js-compat-oai-readme-openai-import
Open

fix(go/plugins/googlegenai): do not resend the cached prefix inline#6141
a2105z wants to merge 2 commits into
genkit-ai:mainfrom
a2105z:docs/js-compat-oai-readme-openai-import

Conversation

@a2105z

@a2105z a2105z commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Context caching stored messages[0..endIndex] on the CachedContent resource but toGeminiContents still sent those same messages in the generate request.
  • The JS plugin slices the cached span off the request; Go did not, so the prefix was billed as cached tokens and as full-price inline input, and the model saw it twice.
  • handleCache now returns the cache boundary. toGeminiContents skips messages at or before that index (-1 means no cache).

Fixes #6137

Test plan

  • Unit test reproduces the bug: with a cache marker on message 0, sending the whole request still includes the prefix
  • Same test asserts the generate contents after the cache boundary are only the follow-up
  • Existing TestToGeminiContents still converts the full request when cachedThrough is -1
  • go test ./go/plugins/googlegenai/
  • CLA check

@github-actions github-actions Bot added docs Improvements or additions to documentation js labels Aug 22, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the README.md for the compat-oai plugin to reflect the new import path for the OpenAI plugin (@genkit-ai/compat-oai/openai) and the use of openAI.model() and openAI.embedder() helpers. It also updates the custom models section to demonstrate using openAICompatible with openrouter and compatOaiModelRef. A review comment suggests removing an unused import z from 'genkit' in the custom models code example.

Comment thread js/plugins/compat-oai/README.md Outdated
},
};
const schema = GenerationCommonConfigSchema.extend({});
import { genkit, z } from 'genkit';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The z object is imported from 'genkit' but is not used anywhere in this code example. We can simplify the import by removing it.

Suggested change
import { genkit, z } from 'genkit';
import { genkit } from 'genkit';

@a2105z a2105z closed this Aug 22, 2026
@a2105z
a2105z deleted the docs/js-compat-oai-readme-openai-import branch August 22, 2026 04:24
handleCache already stores messages[0..endIndex] on the CachedContent
resource. toGeminiContents still sent those same messages in the
generate request, so the prefix was billed twice. Skip the cached span
when building contents, matching the JS plugin.

Fixes genkit-ai#6137
@a2105z
a2105z restored the docs/js-compat-oai-readme-openai-import branch August 22, 2026 04:33
@a2105z a2105z reopened this Aug 22, 2026
@a2105z a2105z changed the title docs(js/plugins/compat-oai): fix OpenAI README import path fix(go/plugins/googlegenai): do not resend the cached prefix inline Aug 22, 2026
@a2105z
a2105z force-pushed the docs/js-compat-oai-readme-openai-import branch from e3168d5 to 179bf2f Compare August 22, 2026 04:33
@github-actions github-actions Bot added the go label Aug 22, 2026
@a2105z

a2105z commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

Hi @genkit-ai/genkit-googlers @pavelgj @apascal07 @huangjeff5 @ifielker @MichaelDoyle @yesudeep @ssbushi @shrutip90 @cabljac @adesinah @xcapaldi — this is the fix for #6137.

Context caching was writing the prefix into CachedContent and still sending it inline. toGeminiContents now skips the cached span, same as JS.

Could someone take a look when you have a moment? Happy to adjust anything. Thank you!

messagesToCache walked the prefix backwards, so after we stop sending
the prefix inline the model would only see a reversed cache.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation go js

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Go] Context caching sends the cached prefix inline as well, doubling prompt cost

1 participant