Skip to content

Fix max_runs check discarding successful tool call in run_until_tool_used#553

Merged
brainlid merged 1 commit into
mainfrom
me-fix-max-runs
May 16, 2026
Merged

Fix max_runs check discarding successful tool call in run_until_tool_used#553
brainlid merged 1 commit into
mainfrom
me-fix-max-runs

Conversation

@brainlid
Copy link
Copy Markdown
Owner

Problem

LLMChain.run_until_tool_used/3 with max_runs: 1 incorrectly errored with Exceeded maximum number of runs (1/1) even when the very first LLM call successfully invoked the target tool. The successful tool result was discarded in favor of the max-runs error.

The bug also affects any max_runs: N configuration where the target tool happens to be called on the Nth attempt — the success is masked by the ceiling check.

Solution

The pipeline in LangChain.Chains.LLMChain.Modes.UntilToolUsed.do_run/2 runs a series of guarded steps where each step either continues with {:continue, chain} or short-circuits with a terminal state. The original order placed check_max_runs immediately after call_llm, so the run counter was incremented and the ceiling check fired before execute_tools and check_until_tool could observe that the target tool had been called.

Reordering moves check_max_runs to run after check_until_tool. Successful termination is detected first; check_max_runs becomes a no-op on already-terminal pipeline states. A comment was added at the call site explaining the ordering constraint so it isn't innocently shuffled in the future.

Changes

  • lib/chains/llm_chain/modes/until_tool_used.ex — Move check_max_runs to run after execute_tools and check_until_tool in do_run/2; add explanatory comment for the ordering constraint.
  • test/chains/llm_chain_test.exs — Add regression test covering max_runs: 1 with a first-call successful tool invocation; asserts the chain returns the ToolResult rather than a max-runs error.

Testing

  • Added a focused regression test in LangChain.Chains.LLMChainTest that mocks ChatOpenAI.call/3 to return the target tool call on the first attempt with max_runs: 1, and asserts both that updated_chain.last_message.role == :tool and that the returned ToolResult is non-error.
  • Existing run_until_tool_used tests (including the pre-existing max-runs exceeded path) continue to cover the failure case where the target tool is never called.

@brainlid brainlid merged commit 9c07a37 into main May 16, 2026
2 checks passed
@brainlid brainlid deleted the me-fix-max-runs branch May 16, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant