fix(ai): ensure that tool calls batched in a single turn always run in parallel

[cdamus]

What it does

Fixes #17533.

AI sub-agent delegations are ordinary tool calls, so whether several delegations issued by a coordinating agent in a single model turn run concurrently was decided entirely by each LLM provider's tool-call execution loop. Those loops were inconsistent: Anthropic and Google already ran a turn's tool calls in parallel, while the OpenAI Chat Completions, OpenAI Response API, Copilot, and Ollama providers ran them sequentially. As a result, parallel delegations (for example the PR reviewer or Architect fanning out explorations) executed one after another on those providers.

This PR makes the contract explicit and uniform: tool calls emitted within a single model response/turn are executed concurrently, since a model serializes genuinely dependent calls across turns instead of batching them.

• Introduces an injectable ToolCallExecutor service in @theia/ai-core as the single common strategy for executing a turn's tool calls. It runs all matched handlers concurrently, applies uniform error handling (tool-not-found and thrown handlers become error results rather than rejections, so one failure never short-circuits its siblings), forwards cancellation, and returns results in input order so providers build their tool-result messages deterministically even though execution interleaves. It is bound in the root container so that downstream applications may rebind it.
• Refactors every provider onto the executor: Anthropic and Google are pure drop-in refactors (they were already concurrent); the OpenAI Response API and Ollama loops are switched from sequential for loops to the executor
• Replaces the OpenAI SDK runTools() runner, whose tool loop is strictly sequential, with a custom streaming/tool loop (ChatCompletionStreamingAsyncIterator) for the OpenAI Chat Completions and Copilot providers, so their turns also execute tool calls concurrently.
• Introduces rebindable <provider>LanguageModelFactory bindings for instantiating the provider language models, so that downstream applications can substitute model implementations.
• Fixes a latent listener leak in ChatResponseImpl: the per-content onDidChange forwarding listener registered by doAddContent was never disposed, and the stream parser re-adds preceding content (e.g. a tool call) on every streamed text token, so a tool call followed by a long text response accumulated one listener per token. This surfaced because functional parallel delegation drives the Explore agents through exactly that pattern.

Caution

Note to reviewers: the first two of these commits I consider to be quite straightforward and safe. The third, which reimplements in parallel the sequential mechanics of a third-party API that the OpenAI and Copilot providers were using, is more risky as it is quite a more complex change. So you might consider carving it off into a separate PR to merge the easier bits first. However, the original bug is not fixed without this change, because anybody using OpenAI ChatGPT model for subagent delegation will not see parallel execution without it: the AI preferences default to the Chat Completions API model implementation, which needs this more complex fix, instead of the Response API model, which relies on the simpler fix.

How to test

Select an OpenAI ChatGPT model (make sure that Response API is not enabled in the preferences) for the PR Reviewer agent and review this PR 😬 . Observe that the subagents looking at different aspects of the PR proceed not in sequence like this:

but in parallel, like this:

and in fact they may even finish out of order, like this:

Then switch the OpenAI preferences to use the Response API and repeat the experiment.

Then repeat the experiment again with as many other models as you have access to.

Also check that you never see "Possible Emitter memory leak detected" warnings in the console during heavy sub-agent streaming.

Alternatively, a more focused experiment that additionally proves the ordering out outputs

The above experiment doesn't conclusively show (necessarily) that the outputs of parallel subagents are collated in the original request/call order. The following alternative scenario does.

Again, for all models that you have access to, and Open AI with and without Response API preference:

1. Assign the model under test to the Architect agent and switch it to Plan Mode (Next) (that variant has no direct read/search tools and must delegate all exploration to the explore agent, and its prompt asks the model to batch delegations). Keep a fast, consistent model on the explore agent.
2. Send a prompt that requires exploring several independent areas at once, e.g.: "Explore these four independent subsystems in parallel, delegating one Explore task per subsystem in a single batch and not waiting between them: packages/terminal, packages/debug, packages/search-in-workspace, packages/notebook. Summarize the main services and entry points of each."
3. Observe that the delegated Explore sub-sessions run concurrently and (probably) complete out of order, while their results are collated back in request order by Architect. Compare wall-clock time to a single delegation. Repeat per provider by swapping the Architect's model.
4. [Optional] Control: ask the Architect to explore the subsystems "one at a time, waiting for each result." Delegations should then run sequentially, confirming an agent can still serialize on purpose.

Also check that you never see "Possible Emitter memory leak detected" warnings in the console during heavy sub-agent streaming.

Follow-ups

• The HuggingFace, Llamafile, and Vercel AI providers were intentionally left out of the executor refactor because Vercel already executes a step's tools concurrently. They could adopt the same pattern later for consistency.

Breaking changes

• This PR introduces breaking changes and requires careful review. If yes, the breaking changes section in the changelog has been updated.

• OpenAiLanguageModelsManagerImpl no longer injects OpenAiModelUtils or OpenAiResponseApiUtils and the openAiModelUtils and responseApiUtils protected fields were removed

• OpenAiModel.createTools() and CopilotLanguageModel.createTools() now return ChatCompletionTool[] instead of RunnableToolFunctionWithoutParse[] because the OpenAI SDK runTools runner is no longer used

• The provider language model classes (AnthropicModel, GoogleModel, OllamaModel, OpenAiModel, CopilotLanguageModel) no longer expose public constructors; instantiate them via the corresponding <provider>LanguageModelFactory instead

• The exported OpenAiModelFactory and OllamaModelFactory symbols were renamed to OpenAiLanguageModelFactory and OllamaLanguageModelFactory

• OpenAiModelUtils moved out of openai-language-model.ts into a new openai-model-utils.ts module

• OpenAiModel and CopilotLanguageModel no longer have a protected toolCallExecutor field; ChatCompletionToolLoopOptions no longer has a toolCallExecutor property

Attribution

None.

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines
• User-facing text is internationalized using the nls service (for details, please see the Internationalization/Localization section in the Coding Guidelines)

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[cdamus]

I myself have tested this PR with:

• Anthropic
• Google
• OpenAI (Chat Completions API)
• OpenAI (Response API)
• Ollama (using qwen3:8b)
• Copilot

[sdirix]

Thank you for the great PR 🎉

• I tested all suggested models and in all of them parallel execution worked ✔️
• I like the introduction of the model factories ✔️
  • Personally I would like to see them go one step further even, i.e. making the models themselves @injectable, but this could be part of a follow up
• I like that we finally get rid of the runTools in OpenAI ✔️

Minor improvements:

• I would remove some of the comments which are specific to the PR but not that important for the code
• I would like to see different names for the new interfaces. See my comment

---

I noticed an issue (in general):

Cancelling parallel tool calls breaks the chat session when using Gemini models. This can be reproduced by instructing parallel code exploration or shell executions, cancelling the session and the trying to continue using the session.

The errors look like this:

Error 400:
Function call is missing a thought_signature in functionCall parts. This is required for tools to work correctly, and missing thought_signature may lead to degraded model performance. Additional data, function call `default_api:delegateToAgent` , position 14. Please refer to https://ai.google.dev/gemini-api/docs/thought-signatures for more details.

Note that I reproduced this is in current Theia as well so it's not a regression of this PR. If you see it in scope, please fix the issue here. If not, then I can create a dedicated ticket for this issue.

[cdamus]

Thanks @sdirix this was really great feedback, so far 😀 I've pushed a merge and a bunch of commits to address your comments. I've also elected to take that further step that you mentioned in refactoring of the creation of the models, to bind the default implementation classes as transient self-services with their dependencies injected. Now the default factories just resolve the model implementations in child containers, widget-style.

I think though that the Gemini cancellation problem had best be raised as a separate issue; it really would be too much scope widening for this PR.

Let me know what you think of these changes!

[sdirix]

Can you resolve the merge conflicts?

[sdirix]

Code looks good and I retested with the Anthropic models. I did not retest all other providers.

[cdamus]

Thanks @sdirix I've resolved merge conflicts in commit 41a1432. They were exclusively related to the server tools feature.

[cdamus]

Hi @eneufeld and @sdirix I've integrated a lot of changes from master branch and resolved conflicts in commit dcbf0ae. The PR Reviewer still seems to be able to delegate in parallel to multiple sub-agents after this reconciliation 😀

[sdirix]

@EclipseSourceAI

[EclipseSourceAI]

Note

Autonomous AI review.

This review was done by an AI agent and therefore may contain mistakes. Feel free to ignore any comment you disagree with. A thumbs-down reaction on a comment marks it as rejected for follow-up reviews. Noting why in a reply helps, since replies are read too.

Resolving all AI comments does not lead to an automatic approval. A maintainer still needs to review and sign off on the overall architecture and design.

To get an updated review after pushing changes, a maintainer may re-request a review from this account.

Running in Eclipse Enclave, submitted via review-guard-mcp

What this PR does

Makes "tool calls emitted in one model turn run concurrently" an explicit, provider-independent contract. A new injectable ToolCallExecutor in @theia/ai-core runs a turn's calls via Promise.all, normalizes errors (tool-not-found and thrown handlers become error results instead of rejections), forwards a cancellation token, and returns outcomes in input order. Anthropic and Google are drop-in refactors; the OpenAI Response API and Ollama loops switch from sequential for loops to the executor; and the OpenAI Chat Completions / Copilot path drops the SDK's runTools() runner in favour of a hand-written ChatCompletionStreamingAsyncIterator over the raw chunk stream. On top of that the provider model classes become @injectable and are created through new <provider>LanguageModelFactory bindings.

Review notes

I compiled the affected packages (tsc --build) and ran their specs: 235 passing, no type errors. I did not verify in the UI, since exercising parallel delegation needs live provider credentials that this run does not have. The author reports manual testing against all six providers and sdirix confirmed parallel execution across the tested models.

Where I would focus:

• ChatCompletionStreamingAsyncIterator is the risky part, as the author flags himself. It reimplements the SDK's tool loop: delta accumulation keyed by index, the assistant/tool message pairing for the next turn, cancellation via controller.abort(). The tests cover the happy paths well. The one concrete gap I found is the "open" tool-call part being emitted before the function name is necessarily known (inline comment), which is unrecoverable downstream because merge never updates the name.
• Executor semantics are sound. Input-order results, one onResult per call, a throwing onResult cannot break the batch, and the not-found/throw cases are covered by tool-call-execution.spec.ts. Ollama is the only caller that does not pass the cancellation token.
• Reuse: the PR extracts AbstractStreamingResponseIterator but leaves ResponseApiToolCallIterator in the same file duplicating that queue verbatim.
• Breaking changes: the CHANGELOG list is good but incomplete (two removed exported symbols plus the OpenAiModelUtils module move). Also the PR description is now stale: it says ToolCallExecutor is connection-scoped and lists an OpenAiResponseApiUtils scope change, but the final code binds both in the root container.
• The console.* to ILogger conversions across the provider models are drive-by, though they follow naturally from making the models injectable. A maintainer should decide whether that is acceptable scope here.

Nothing I found looks like a merge blocker.

[cdamus]

Commit 395fa57 addresses all of the EclipseSourceAI bot's comments, plus an additional problem that I found myself in the OpenAI chat completion stream: cancellation did not promptly terminate the iterator while tools execute.

[EclipseSourceAI]

Note

Autonomous AI review.

This review was done by an AI agent and therefore may contain mistakes. Feel free to ignore any comment you disagree with. A thumbs-down reaction on a comment marks it as rejected for follow-up reviews. Noting why in a reply helps, since replies are read too.

Resolving all AI comments does not lead to an automatic approval. A maintainer still needs to review and sign off on the overall architecture and design.

To get an updated review after pushing changes, a maintainer may re-request a review from this account.

Running in Eclipse Enclave, submitted via review-guard-mcp

Follow-up review. Only one commit landed since my last pass (395fa57), and it's entirely a response to that review plus one extra fix the author found himself (prompt cancellation could stall behind an uninterruptible tool handler in ChatCompletionStreamingAsyncIterator, or let a late tool result leak out after cancellation). I traced through cancel()/handleIncoming()/dispose() and the new spec cases (including the two cancellation-race tests) and the fix is sound: done is checked before delivering any part, dispose() is idempotent, and the executor's cancellation token still reaches Ollama's tool handlers via ToolInvocationContext.

No new findings from this pass.

These previous comments can be resolved as they are now handled:

• Open tool-call part could stay nameless if the name streamed after the id
• ResponseApiToolCallIterator duplicated the queue now extracted into AbstractStreamingResponseIterator
• Ollama didn't forward the cancellation token/ToolInvocationContext into tool execution
• ChatCompletionStreamingAsyncIterator should inject ToolCallExecutor directly instead of threading it through options
• CHANGELOG was missing several breaking API changes
• as any cast in ollama package.spec.ts, processToolCalls should be protected

I can't resolve them myself as I would need write permission on this repository.

[sdirix]

The changelog entries are now waaaaaaay too long and extensive for my taste.

[cdamus]

The changelog entries are now waaaaaaay too long and extensive for my taste.

Thanks! I've resolved conflicts and tried to reduce the verbosity of the changelog, which was helped by all of my Copilot changes having been obviated by incoming feature work from master (all of the tool calling is now handled within the Copilot CLI, so no longer our responsibility).

[EclipseSourceAI]

Note

Autonomous AI review.

This review was done by an AI agent and therefore may contain mistakes. Feel free to ignore any comment you disagree with. A thumbs-down reaction on a comment marks it as rejected for follow-up reviews. Noting why in a reply helps, since replies are read too.

Resolving all AI comments does not lead to an automatic approval. A maintainer still needs to review and sign off on the overall architecture and design.

To get an updated review after pushing changes, a maintainer may re-request a review from this account.

Running in Eclipse Enclave, submitted via review-guard-mcp

Follow-up review. Since my last pass (commit 395fa57) the only thing that landed is the merge from master (commit 7802c24).

Checked the diff: ai-core, ai-openai, ai-anthropic, ai-google and ai-ollama have zero changes since 395fa57, so the code I already reviewed there is untouched. The CHANGELOG.md entries for this PR got reordered/reworded during conflict resolution but the content is the same, still complete and accurate about the breaking changes. ai-copilot is no longer touched by this PR at all: the Copilot CLI migration from master (#17919) replaced the REST-based CopilotLanguageModel this PR used to touch, so the tool-call concurrency question for Copilot is now handled entirely inside the CLI, outside this PR's scope, matching what the author said in the PR thread.

No new findings from this pass. All threads I raised earlier are already marked resolved.