Skip to content

test(llm): verify quota retry policy in async completions (#4917 follow-up) - #27

Closed
enyst wants to merge 1 commit into
review/pr-4917-quota-fallbackfrom
fix/pr-4917-quota-review-20260910
Closed

enyst wants to merge 1 commit into
review/pr-4917-quota-fallbackfrom
fix/pr-4917-quota-review-20260910

Conversation

@enyst

@enyst enyst commented Sep 10, 2026

Copy link
Copy Markdown
Owner

HUMAN:


AGENT:
Prepared by an AI agent on behalf of Engel as a follow-up to OpenHands#4917.

Why

The latest parent revision already fixes the broad quota match and the type annotation raised in review, but its quota regressions only exercised synchronous completion.

Summary

  • Exercise both usage_limit_reached and insufficient_quota through synchronous and asynchronous completion.
  • Verify that transient Vertex-style quota errors still retry before fallback in both paths.
  • Assert that the successful or final attempted provider is the configured fallback.

Issue Number

Follow-up to OpenHands#4917; related issue OpenHands#4936.

How to Test

From this branch:

make build
uv run pytest tests/sdk/llm/test_exception_classifier.py tests/sdk/llm/test_llm_fallback.py -q
uv run pre-commit run --files tests/sdk/llm/test_llm_fallback.py
git diff --check

Result: 45 passed and all applicable pre-commit hooks passed.
The tests drive the public LLM.completion() and LLM.acompletion() paths, injecting provider failures at the LiteLLM transport boundary.
Hard quota errors make one primary attempt and then use the fallback; transient quota errors make two primary attempts before trying the fallback.
No live provider quota was consumed or deliberately exhausted.

Video/Screenshots

Not applicable; this is regression coverage for backend retry behavior.

Design Doc

Not applicable; no production behavior or public API changes.

Type

  • Docs / chore

Notes

This PR targets review/pr-4917-quota-fallback, an exact mirror of original PR OpenHands#4917 at 2b97474214e81ee1e648462350e1884920633e82, and contains only the additional regression coverage.

Creating the PR directly against all-hands-bot/software-agent-sdk:quota-fallback-immediate was rejected by GitHub with HTTP 403 (Resource not accessible by integration), so this fork PR makes the change reviewable now; merging it here does not update the original PR branch.
The enyst SDK fork's main branch was fast-forwarded to upstream before preparing this change.

Exercise both hard quota codes and transient provider quota retries through synchronous and asynchronous completion.

Co-authored-by: openhands <openhands@all-hands.dev>
@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@enyst

enyst commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Relocated upstream: the regression test is now on the OpenHands#4917 branch (all-hands-bot:quota-fallback-immediate), so OpenHands#4917 carries it. Closing this fork mirror. — smolpaws for Engel 🐾

@enyst enyst closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant