Skip to content

fix(sdk): tolerate missing cache fields in token usage buckets - #5169

Open
carvalab wants to merge 1 commit into
OpenHands:mainfrom
carvalab:fix/cache-buckets-getattr
Open

carvalab wants to merge 1 commit into
OpenHands:mainfrom
carvalab:fix/cache-buckets-getattr

Conversation

@carvalab

@carvalab carvalab commented Sep 18, 2026

Copy link
Copy Markdown

HUMAN:

I saved a MiniMax profile in Agent Canvas settings and chatted in a new conversation. Also chatted with ZAI glm-5.3-flash. All work with the fix.


AGENT:

Why

Any provider whose usage response includes prompt_tokens_details without cache fields crashes profile validation and token recording with 'PromptTokensDetailsWrapper' object has no attribute 'cache_creation_tokens'. Reproduced with MiniMax in Agent Canvas settings; other reporters hit it with different providers (see #5099).

Root cause: Telemetry._cache_buckets read details.cache_creation_tokens directly, guarded only by "cache_creation_tokens" in details.model_fields_set. LiteLLM's PromptTokensDetailsWrapper.__init__ deletes unset optional fields, while its __setattr__ mirroring keeps the name in model_fields_set. So the guard passes and the read raises AttributeError. I confirmed this against the installed SDK: model_fields_set contained the name even though the attribute was gone.

Summary

  • Read both cache namings (cache_creation_tokens, cache_write_tokens) through getattr with a 0 default in _cache_buckets, matching how the rest of the file already reads optional usage fields.
  • Added regression test test_record_usage_with_real_details_no_cache_fields using a real PromptTokensDetailsWrapper (the existing MagicMock-based tests masked this bug).

Issue Number

Fixes #5168.

How to Test

Reproduced and verified against the installed stack (openhands-sdk 1.49.1, Agent Canvas via npm exec @openhands/agent-canvas):

  1. Reproduced the crash with the installed interpreter: built a Usage with PromptTokensDetailsWrapper(cached_tokens=0), which is what MiniMax returns, and called Telemetry._cache_buckets. Result before the fix: AttributeError: 'PromptTokensDetailsWrapper' object has no attribute 'cache_creation_tokens'.
  2. Applied the fix and re-ran the same script: MiniMax-like usage returns (0, 0), usage with cache write returns (5, 7), usage without details returns (0, 0).
  3. Ran the full tests/sdk/llm/test_llm_telemetry.py file: 43 passed. The new test fails on the old code and passes with the fix.
  4. Patched the installed Agent Canvas backend the same way, restarted it, and saved a MiniMax profile (minimax/MiniMax-M3, https://api.minimax.io/v1) in Settings → LLM: validation passed where it previously failed with the AttributeError, and a new conversation with MiniMax answers normally. Also verified with ZAI glm-5.3-flash in a new conversation.

Video/Screenshots

No UI change, so no screenshots. Evidence is the reproduction output and test run described above.

Design Doc

Not needed, this is a small bug fix.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

The model_fields_set guard looked safe but is not: LiteLLM's __setattr__ mirroring adds the mirrored name to model_fields_set before __init__ deletes both attributes. Later reads of optional wrapper fields should always use getattr with a default.

Jev-Fast-Audit

Jev fast audit · estimates · 0.37s · commit 460d8c2
Strongest signal: No primary concern selected.
Evidence: No primary concern to locate.
Coverage: complete supplied coverage; 2/2 hunks, 2/2 files.

All estimates and evidence
Estimate Likelihood / value Direct evidence
SQL injection 3.0% No direct hunk selected
Command injection 2.0% No direct hunk selected
Weakened authentication 3.0% No direct hunk selected
Weakened authorization 3.0% No direct hunk selected
Contract regression 9.0% No direct hunk selected
Data loss 3.0% No direct hunk selected
Sensitive data disclosure 3.0% No direct hunk selected
Unexpected data transfer 2.0% No direct hunk selected
Credential misuse 3.0% No direct hunk selected
Untrusted instruction authority 2.0% No direct hunk selected
Package source redirection 3.0% No direct hunk selected
Unverified remote execution 2.0% No direct hunk selected
Privileged environment access 2.0% No direct hunk selected
Security assessment bypass 2.0% No direct hunk selected
Prohibited workload 2.0% No direct hunk selected
Primary concern None selected; confidence 91.0% No primary concern to locate

@all-hands-bot

Copy link
Copy Markdown
Collaborator

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

🚦 CI is currently failing on this PR's latest commit.

Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request @all-hands-bot as a reviewer to have it reviewed regardless of CI status.)

This is an automated check - no AI was used to generate this comment.

@atifnaqvi-jpg

Copy link
Copy Markdown

Corroborating reproduction — this fires in profile pre-flight, not just during a run.

Environment: fresh @openhands/agent-canvas@1.22.0, agent-server 1.49.4, automation 1.14.0, single-core arm64 VM, Ubuntu 24.04. Clean state (no inherited settings or conversations).

Provider: Ollama-hosted model via the OpenAI-compatible route — profile with Custom Model: openai/deepseek-v4.1-flash:cloud and Base URL: https://ollama.com/v1.

Symptom: the profile Save fails with the AttributeError, even though the underlying model call succeeds.

Log:

18:04:46  [agent-server] /home/ubuntu/.cache/uv/archive-v0/Flx_gn5IYzJn0qZZ/lib/python3.12/site-packages/openhands/sdk/llm/utils/telemetry.py:363: UserWarning: Cost calculation failed: This model isn't mapped yet. model=deepseek-v4.1-flash:cloud, custom_llm_provider=openai.
18:04:46  [agent-server] INFO  Profile 'ollama_deepseek-v4.1-flash' pre-flight failed (unknown): 'PromptTokensDetailsWrapper' object has no attribute 'cache_creation_tokens'  profiles_router.py:360
18:05:06  [agent-server] INFO  Profile 'ollama_deepseek-v4.1-flash' pre-flight failed (unknown): 'PromptTokensDetailsWrapper' object has no attribute 'cache_creation_tokens'  profiles_router.py:360
18:07:17  [agent-server] INFO  Profile 'ollama_deepseek-v4.1-flash' pre-flight failed (unknown): 'PromptTokensDetailsWrapper' object has no attribute 'cache_creation_tokens'  profiles_router.py:360
18:07:22  [agent-server] INFO  Profile 'ollama_deepseek-v4.1-flash' pre-flight failed (unknown): 'PromptTokensDetailsWrapper' object has no attribute 'cache_creation_tokens'  profiles_router.py:360
18:07:33  [agent-server] INFO  Profile 'ollama_deepseek-v4.1-flash' pre-flight failed (unknown): 'PromptTokensDetailsWrapper' object has no attribute 'cache_creation_tokens'  profiles_router.py:360

Why this is a distinct case: the same profile reaches the model successfully — routing errors show up differently and are absent here (a wrong base URL produces path "/v1/api/generate" not found, and an earlier attempt produced path "/api/chat/completions" not found). Once routing is correct, the call completes and the failure moves to usage/token processing. So the crash is independent of the endpoint and the model id, and it occurs after a successful response.

Also confirmed against the installed SDK: PromptTokensDetailsWrapper(cached_tokens=0) raises on cache_creation_tokens, while PromptTokensDetailsWrapper(cached_tokens=0, cache_creation_input_tokens=0) returns 0 — matching the root cause described above (the name is present in model_fields_set but the attribute was deleted).

Reporter context: this blocks saving any profile for a provider that omits cache fields, and pre-flight is the only validation path in the Agent Canvas settings UI — so there is no way to persist such a profile at all, independently of whether conversations would run.

Happy to re-test against a patched build.

@openhands-ai

openhands-ai Bot commented Sep 22, 2026

Copy link
Copy Markdown

I'm on it! atifnaqvi-jpg can track my progress at all-hands.dev

atifnaqvi-jpg commented Sep 22, 2026

Copy link
Copy Markdown

Verified the fix and the reported pre-flight reproduction. Two real blockers remain on the current head (460d8c2) — both are CI failures, not nitpicks — plus the regression test does not actually guard the bug in CI's pinned environment.

1. getattr fix is rejected by the repo's own pre-commit hook

openhands-sdk/.../utils/telemetry.py now contains four new getattr calls, which the Forbid dynamic attribute access in SDK hook (scripts/check_forbidden_dynamic_attributes.py) rejects. The baseline is append-only-shrinking (--baseline-ref allows it only to shrink), so this cannot be baselined away:

$ uv run python scripts/check_forbidden_dynamic_attributes.py
openhands-sdk/openhands/sdk/llm/utils/telemetry.py:262: forbidden dynamic attribute call: getattr
openhands-sdk/openhands/sdk/llm/utils/telemetry.py:262: forbidden dynamic attribute call: getattr
openhands-sdk/openhands/sdk/llm/utils/telemetry.py:265: forbidden dynamic attribute call: getattr

Once the Pre-commit checks run executes (it currently shows action_required, awaiting approval), this step fails and fails the job. Reading through model_dump() achieves the same thing with no dynamic attribute access (pydantic model_dump() omits fields that were deleted, so a null/absent cache_creation_tokens is simply not in the dict):

dumped = details.model_dump()
cache_read = dumped.get("cached_tokens") or 0
cache_write = (
    dumped.get("cache_creation_tokens") or dumped.get("cache_write_tokens") or 0
)
return int(cache_read), int(cache_write)

2. ruff format fails on the same file

The return-statement wrapping does not match the repo's formatter, so Ruff format (pre-commit) fails independently of §1. uv run ruff format fixes it.

3. The new regression test passes on the old code under CI's pinned LiteLLM

The claim "the new test fails on the old code" does not hold in CI. uv.lock pins litellm==1.93.0 and CI runs uv sync --frozen, and under 1.93.0 a real PromptTokensDetailsWrapper(cached_tokens=25) does not put cache_creation_tokens in model_fields_set, so the old guard is False and it returns 0 without raising:

A direct cached-only (MiniMax): OLD=(25, 0) NEW=(25, 0)   # old code does NOT crash

(That version-dependent __setattr__/model_fields_set behavior is exactly your root-cause analysis — it's just not observable at 1.93.0. It does reproduce at 1.95.0+, which is what the reporter's un-pinned litellm>=1.93.0 resolves to, so the user-facing bug is real.)

Two added shapes do fail on the old code even at 1.93.0, so the guard is version-independent:

B dict {"cached_tokens": 25, "cache_creation_tokens": None}:  OLD=RAISED AttributeError  NEW=(25, 0)
C wrapper(cached_tokens=2, cache_write_tokens=5):             OLD=(2, 0)                 NEW=(2, 5)

4. Pre-existing MagicMock test also masked this

test_record_usage_with_cache_read stubs prompt_tokens_details with a MagicMock, which answers any attribute access, so it cannot catch the deleted-field behavior. Swapping it for a real wrapper (as the PR already argues for) makes it meaningful; note that a model_dump()-based fix otherwise breaks it, because MagicMock().model_dump() returns another MagicMock.

Verified fix

I applied a policy-compliant version of the fix locally and confirmed:

  • uv run pre-commit run --files <both files> → all hooks pass (ruff format/lint, pycodestyle, pyright, forbidden-dynamic-attributes).
  • tests/sdk/llm/test_llm_telemetry.py → 44 passed; the two strengthened tests fail on the unmodified main code and pass with the fix.
  • Behavior across litellm 1.93.0 / 1.95.0 / 1.99.0 is identical for the four shapes above.

Could not push

Head is carvalab:fix/cache-buckets-getattr, and the available token account only has read access to that fork (push: false), so I could not push the update myself — posting the diagnosis and patch instead. Happy to hand off a branch if someone grants write access.

This message was created by an AI agent (OpenHands) on behalf of atifnaqvi-jpg.

Patch (applies on top of 460d8c2)
From 3b7d404b1c11f47d9b54a2f6de64f8adb43e9fb6 Mon Sep 17 00:00:00 2001
From: openhands <openhands@all-hands.dev>
Date: Tue, 22 Sep 2026 18:44:04 +0000
Subject: [PATCH] fix(sdk): read cache buckets via model_dump to satisfy SDK
 getattr policy

The previous fix read cache fields with getattr, which the repo's
'Forbid dynamic attribute access in SDK' pre-commit hook rejects (the
baseline may only shrink), so pre-commit/CI failed.

Read prompt_tokens_details through model_dump() instead: it exposes only
fields that still exist and accepts either cache-write naming, with no
dynamic attribute calls. Also

- strengthen the regression tests: cover the explicit
  'cache_creation_tokens: null' shape (reproduces the AttributeError on
  the LiteLLM version pinned in uv.lock, where a cached_tokens-only
  wrapper does not) and the cache_write_tokens naming;
- replace the MagicMock-based cache-read test with a real
  PromptTokensDetailsWrapper, since a mock answers every attribute
  access and hides the deleted-optional-field behavior.

Co-authored-by: openhands <openhands@all-hands.dev>
---
 .../openhands/sdk/llm/utils/telemetry.py      | 22 ++++---
 tests/sdk/llm/test_llm_telemetry.py           | 58 ++++++++++++++-----
 2 files changed, 57 insertions(+), 23 deletions(-)

diff --git a/openhands-sdk/openhands/sdk/llm/utils/telemetry.py b/openhands-sdk/openhands/sdk/llm/utils/telemetry.py
index fbfd94a..50dc625 100644
--- a/openhands-sdk/openhands/sdk/llm/utils/telemetry.py
+++ b/openhands-sdk/openhands/sdk/llm/utils/telemetry.py
@@ -255,16 +255,20 @@ class Telemetry(BaseModel):
             details = usage.prompt_tokens_details
             if details is None:
                 return 0, 0
-            # NB: PromptTokensDetailsWrapper deletes unset optional fields
-            # (e.g. providers without prompt caching like MiniMax), so direct
-            # attribute access raises AttributeError. getattr with a default
-            # is the only safe read here.
-            cache_write = getattr(details, "cache_creation_tokens", 0) or getattr(
-                details, "cache_write_tokens", 0
-            )
-            return int(getattr(details, "cached_tokens", 0) or 0), int(
-                cache_write or 0
+            # NB: PromptTokensDetailsWrapper deletes unset optional fields, so a
+            # direct read raises AttributeError even when the name is still in
+            # ``model_fields_set`` (its ``__setattr__`` mirroring repopulates the
+            # set while ``__init__`` deletes the attribute). Reading through
+            # ``model_dump`` returns only the fields that actually exist, which
+            # also lets us accept either cache-write naming.
+            dumped = details.model_dump()
+            cache_read = dumped.get("cached_tokens") or 0
+            cache_write = (
+                dumped.get("cache_creation_tokens")
+                or dumped.get("cache_write_tokens")
+                or 0
             )
+            return int(cache_read), int(cache_write)
 
         details = usage.input_tokens_details
         cache_read = details.cached_tokens if details is not None else 0
diff --git a/tests/sdk/llm/test_llm_telemetry.py b/tests/sdk/llm/test_llm_telemetry.py
index bec2d17..521e2c4 100644
--- a/tests/sdk/llm/test_llm_telemetry.py
+++ b/tests/sdk/llm/test_llm_telemetry.py
@@ -166,13 +166,16 @@ class TestTelemetryTokenUsage:
 
     def test_record_usage_with_cache_read(self, basic_telemetry):
         """Test token usage recording with cache read tokens."""
-        # Create a mock usage with prompt_tokens_details
-        usage = Usage(prompt_tokens=100, completion_tokens=50, total_tokens=150)
+        from litellm.types.utils import PromptTokensDetailsWrapper
 
-        # Mock the prompt_tokens_details attribute
-        mock_details = MagicMock()
-        mock_details.cached_tokens = 25
-        usage.prompt_tokens_details = mock_details
+        # Use a real wrapper: a MagicMock answers every attribute access, which
+        # hides the "optional field deleted" behavior this code must tolerate.
+        usage = Usage(
+            prompt_tokens=100,
+            completion_tokens=50,
+            total_tokens=150,
+            prompt_tokens_details=PromptTokensDetailsWrapper(cached_tokens=25),
+        )
 
         basic_telemetry._record_usage(usage, "test-id", 4096)
 
@@ -195,20 +198,23 @@ class TestTelemetryTokenUsage:
         token_usage = basic_telemetry.metrics.token_usages[0]
         assert token_usage.cache_write_tokens == 30
 
-    def test_record_usage_with_real_details_no_cache_fields(self, basic_telemetry):
-        """Providers without prompt caching (e.g. MiniMax) return a real
-        PromptTokensDetailsWrapper with the cache attributes deleted.
+    def test_record_usage_with_null_cache_creation_tokens(self, basic_telemetry):
+        """A provider that emits ``cache_creation_tokens: null`` leaves the key
+        in ``model_fields_set`` while the wrapper deletes the attribute.
 
-        Regression test: _record_usage must not raise AttributeError here,
-        and must still count cache reads.
+        Regression test for the exact AttributeError reported in #5168: the old
+        ``"cache_creation_tokens" in details.model_fields_set`` guard passed, so
+        the direct attribute read raised. Reproducible on every LiteLLM version,
+        including the one pinned in ``uv.lock``.
         """
-        from litellm.types.utils import PromptTokensDetailsWrapper
-
         usage = Usage(
             prompt_tokens=100,
             completion_tokens=50,
             total_tokens=150,
-            prompt_tokens_details=PromptTokensDetailsWrapper(cached_tokens=25),
+            prompt_tokens_details={
+                "cached_tokens": 25,
+                "cache_creation_tokens": None,
+            },
         )
 
         basic_telemetry._record_usage(usage, "test-id", 4096)
@@ -217,6 +223,30 @@ class TestTelemetryTokenUsage:
         assert token_usage.cache_read_tokens == 25
         assert token_usage.cache_write_tokens == 0
 
+    def test_record_usage_with_cache_write_tokens_naming(self, basic_telemetry):
+        """OpenAI-compatible backends (e.g. kimi-k2) report cache writes as
+        ``cache_write_tokens`` rather than ``cache_creation_tokens``.
+
+        Regression test: both namings must be honored, otherwise cache writes
+        from these providers are silently recorded as 0.
+        """
+        from litellm.types.utils import PromptTokensDetailsWrapper
+
+        usage = Usage(
+            prompt_tokens=100,
+            completion_tokens=50,
+            total_tokens=150,
+            prompt_tokens_details=PromptTokensDetailsWrapper(
+                cached_tokens=2, cache_write_tokens=5
+            ),
+        )
+
+        basic_telemetry._record_usage(usage, "test-id", 4096)
+
+        token_usage = basic_telemetry.metrics.token_usages[0]
+        assert token_usage.cache_read_tokens == 2
+        assert token_usage.cache_write_tokens == 5
+
     def test_record_usage_missing_tokens(self, basic_telemetry):
         """Test token usage recording with missing token counts."""
         usage = Usage()  # Empty usage
-- 
2.47.3

@openhands-ai

openhands-ai Bot commented Sep 22, 2026

Copy link
Copy Markdown

I've completed the review. Here's my summary.

What I found

The PR's diagnosis is correct and the user-facing bug is real (the reporter's un-pinned litellm>=1.93.0 resolves to a version where PromptTokensDetailsWrapper leaves cache_creation_tokens in model_fields_set while deleting the attribute). But the fix as submitted had two genuine CI blockers and a test that doesn't actually guard the bug:

  1. getattr is banned in SDK source. The repo's Forbid dynamic attribute access in SDK pre-commit hook rejected the four new getattr calls, and its baseline may only shrink, so it can't be waived. I rewrote the read to go through model_dump(), which omits deleted fields and needs no dynamic attribute access.

  2. ruff format fails on the same file's wrapped return statement.

  3. The regression test doesn't fail on the old code under CI's environment. uv.lock pins litellm==1.93.0 and CI runs uv sync --frozen; at 1.93.0 a cached_tokens-only wrapper does not trigger the old crash. I added two shapes that do fail on the old code at any version: an explicit cache_creation_tokens: null (reproduces the exact AttributeError) and the cache_write_tokens naming (old code silently recorded 0).

  4. A pre-existing MagicMock-based test answered every attribute access and masked the deleted-field behavior, so I switched it to a real wrapper (a model_dump() fix otherwise breaks it, since MagicMock().model_dump() returns another mock).

Verified with uv run pre-commit run --files <both> (all hooks pass), tests/sdk/llm/test_llm_telemetry.py (44 passed; strengthened tests fail on unmodified main and pass with the fix), and identical behavior across litellm 1.93.0 / 1.95.0 / 1.99.0.

Blocker I could not clear

The PR head is carvalab:fix/cache-buckets-getattr, and the available GITHUB_TOKEN account (atifnaqvi-jpg) has only read access to that fork (push: false), so git push returns 403. My commit (3b7d404) is ready locally but unpushed.

I posted the full diagnosis plus a ready-to-apply patch to PR #5169.

Next step needed from you: either grant this account write access to carvalab/software-agent-sdk (or tell me the correct fork/remote to push to) so I can update the PR branch directly, or apply the attached patch yourself.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

The diagnosis is correct and the user-facing bug is real, but this head cannot pass the repository's own gates and the added regression test does not guard the bug in the pinned environment. Everything below was reproduced at 460d8c2 with the repo's own toolchain (uv sync --dev, litellm==1.93.0 from uv.lock).

1. Blocking — the new getattr calls violate the SDK's dynamic-attribute hook.

uv run python scripts/check_forbidden_dynamic_attributes.py exits 1 with three violations: telemetry.py:262 (x2) and telemetry.py:265. That hook is wired in .pre-commit-config.yaml as Forbid dynamic attribute access in SDK (always_run: true, pass_filenames: false), and the committed baseline may only shrink (--baseline-ref rejects additions), so this cannot be baselined away. It also contradicts the repository's stated preference for typed access over getattr/hasattr guards. Reading the details through details.model_dump() gives the same tolerance with no dynamic attribute access, because a deleted field is simply absent from the dump.

2. Blocking — ruff format fails on the same file.

uv run ruff format --check reports Would reformat, and the Ruff format pre-commit hook rewrites the file and fails. The offending part is the wrapping of the return statement on the new lines 265-267.

3. The new regression test does not guard the bug at the pinned LiteLLM.

I restored the pre-fix _cache_buckets in memory only (no tracked file was modified) and ran the new test: it passes. Under litellm==1.93.0, PromptTokensDetailsWrapper(cached_tokens=25) produces model_fields_set == {'cached_tokens'}, so the old guard is False and returns (25, 0) without raising. The same holds for PromptTokensDetailsWrapper(cached_tokens=0), the shape named in the PR's "How to Test" — so both the "fails on the old code" claim and that reproduction do not hold in the environment CI installs via uv sync --frozen.

Shapes that do raise on the old code even at 1.93.0, and would make the test a genuine guard: prompt_tokens_details={"cached_tokens": 25, "cache_creation_tokens": None}, or the equivalent raw ModelResponse usage dict with cache_creation_tokens: null (both keep the name in model_fields_set while the attribute is deleted).

4. The fix target has moved, and the branch no longer applies cleanly.

This PR is CONFLICTING with main (mergeable_state: dirty; git merge-tree reports a content conflict in telemetry.py). Main has since replaced the _cache_buckets guard with a typed normalize_usage() adapter (#5029) that reintroduces the same unsafe pattern: "cache_creation_tokens" in prompt_details.model_fields_set followed by a direct prompt_details.cache_creation_tokens read. I confirmed that this raises AttributeError on main for the cache_creation_tokens: null shape. The fix needs to be rebased and applied inside normalize_usage, not only _cache_buckets.

One scope note for the maintainer: this change also starts counting cache_write_tokens (the kimi-k2 naming) that the previous code ignored. That looks correct (litellm's own cost calculator reads both names) but it is a token-accounting change beyond the reported crash.

CI note: on this fork head the Run tests and Pre-commit checks runs are all action_required, so there is no live CI result to read; the hook failures above were reproduced locally with the pinned toolchain.

🔄 CHANGES REQUESTED

# (e.g. providers without prompt caching like MiniMax), so direct
# attribute access raises AttributeError. getattr with a default
# is the only safe read here.
cache_write = getattr(details, "cache_creation_tokens", 0) or getattr(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: these getattr calls fail the repository's own Forbid dynamic attribute access in SDK pre-commit hook (scripts/check_forbidden_dynamic_attributes.py exits 1 with violations at 262 x2 and 265), and the committed baseline may only shrink, so this cannot be allowed through. uv run ruff format --check also reports this file needs reformatting.

model_dump() achieves the same tolerance without dynamic attribute access — deleted optional fields are simply absent from the dump:

            dumped = details.model_dump()
            cache_read = dumped.get("cached_tokens") or 0
            cache_write = (
                dumped.get("cache_creation_tokens")
                or dumped.get("cache_write_tokens")
                or 0
            )
            return int(cache_read), int(cache_write)

Note this would also require replacing the MagicMock in test_record_usage_with_cache_read, since a mock's model_dump() returns another mock.

token_usage = basic_telemetry.metrics.token_usages[0]
assert token_usage.cache_write_tokens == 30

def test_record_usage_with_real_details_no_cache_fields(self, basic_telemetry):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test does not fail on the pre-fix code in the environment CI installs. With the old _cache_buckets restored in memory, this test passes: at litellm==1.93.0 (pinned in uv.lock) PromptTokensDetailsWrapper(cached_tokens=25) yields model_fields_set == {'cached_tokens'}, so the old guard is False and returns (25, 0) without raising. PromptTokensDetailsWrapper(cached_tokens=0) behaves the same way, so it cannot stand in as the reproduction either.

A shape that does raise on the old code at 1.93.0 is the explicit null, which keeps the name in model_fields_set while the attribute is deleted:

        usage = Usage(
            prompt_tokens=100,
            completion_tokens=50,
            total_tokens=150,
            prompt_tokens_details={"cached_tokens": 25, "cache_creation_tokens": None},
        )

Covering that (or the equivalent raw ModelResponse usage dict with cache_creation_tokens: null) makes this a real regression guard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AttributeError in Telemetry._cache_buckets for providers without prompt caching

4 participants