fix(go/plugins/googlegenai): do not resend the cached prefix inline - #6141
fix(go/plugins/googlegenai): do not resend the cached prefix inline#6141a2105z wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the README.md for the compat-oai plugin to reflect the new import path for the OpenAI plugin (@genkit-ai/compat-oai/openai) and the use of openAI.model() and openAI.embedder() helpers. It also updates the custom models section to demonstrate using openAICompatible with openrouter and compatOaiModelRef. A review comment suggests removing an unused import z from 'genkit' in the custom models code example.
| }, | ||
| }; | ||
| const schema = GenerationCommonConfigSchema.extend({}); | ||
| import { genkit, z } from 'genkit'; |
handleCache already stores messages[0..endIndex] on the CachedContent resource. toGeminiContents still sent those same messages in the generate request, so the prefix was billed twice. Skip the cached span when building contents, matching the JS plugin. Fixes genkit-ai#6137
e3168d5 to
179bf2f
Compare
|
Hi @genkit-ai/genkit-googlers @pavelgj @apascal07 @huangjeff5 @ifielker @MichaelDoyle @yesudeep @ssbushi @shrutip90 @cabljac @adesinah @xcapaldi — this is the fix for #6137. Context caching was writing the prefix into Could someone take a look when you have a moment? Happy to adjust anything. Thank you! |
messagesToCache walked the prefix backwards, so after we stop sending the prefix inline the model would only see a reversed cache.
Summary
messages[0..endIndex]on theCachedContentresource buttoGeminiContentsstill sent those same messages in the generate request.handleCachenow returns the cache boundary.toGeminiContentsskips messages at or before that index (-1means no cache).Fixes #6137
Test plan
TestToGeminiContentsstill converts the full request whencachedThroughis-1go test ./go/plugins/googlegenai/