feat(ai-copilot): replace the REST transport with the Copilot CLI

[ndoschek]

What it does

@theia/ai-copilot now serves all requests through the official GitHub Copilot CLI instead of the
Copilot REST API. GitHub grants access to the Copilot models per OAuth application, and Theia's is not
entitled for the current lineup, so the REST path only ever exposed a small legacy subset regardless of
the user's subscription. The CLI is entitled, so routing through it makes the current models available.

• The sign-in is driven from within the application (copilot login --device-code) and keeps using the
  existing dialog.
• The credentials belong to the application: the login runs against a private Copilot home, its token is
  taken over into Theia's credential store, and signing out removes that entry. A token in the
  environment or a GitHub CLI sign-in is not used, so neither can make Theia appear signed in.
• Tool calling is bridged into the CLI session, so Copilot models can drive the agents.
• The prompt of a Theia agent becomes the system message of the session, replacing the agent
  instructions of the CLI.
• The runtime runs without the ambient behaviour of the CLI: only the tools of the request are
  available, it uses a Copilot home below Theia's configuration directory, the session of a request is
  deleted once answered, and the CLI of a frontend connection is stopped when it closes.
• The integration is marked experimental, and its preferences carry the experimental tag.
• Removed: the OAuth app sign-in, CopilotOAuthConfig, CopilotLanguageModel and the openai
  dependencies. See the breaking changes and doc/Migration.md.

The CLI is a prerequisite on the backend host rather than something the application ships. Users install
it with npm install -g @github/copilot, and it is located through the new
ai-features.copilot.executablePath preference, COPILOT_CLI_PATH, the installation of the
application, the PATH and the global npm directory.

Neither the CLI nor @github/copilot-sdk is a dependency of the extension, since the SDK package pulls
in the CLI as a large platform-specific binary that would land in the tree and lockfile of every
application. The SDK is loaded at runtime from the CLI that serves the requests, and the part of its API
that is used is mirrored in copilot-sdk-types.ts, with a scheduled workflow checking the mirror
against the released SDK.

Many thanks to @jdubois, whose #17629 introduced the SDK transport as an opt-in path and showed that this
is the way to reach the current models. This builds directly on that work: his copilot-sdk-mappers.ts
is kept and extended (session system message, de-duplication of dated model releases), while the client
provider and language model are reworked around the new authentication, tool handling and CLI lookup.

Kept as a draft until I have tested it on Windows: the CLI lookup has a Windows-specific path (a global
npm install puts a copilot.cmd shim on the PATH, and the npm root -g fallback goes through a
shell) that is covered by unit tests but not by a real run. Feedback on the approach is welcome
meanwhile.

How to test

Requires an active GitHub Copilot subscription and the locally installed copilot CLI: npm install -g @github/copilot.

1. Sign in, and check which application you authorize

1. Run Copilot: Sign In, or click the Copilot status bar entry.
2. The dialog shows a device code and a link. No browser should open by itself, the code has to stay
   readable and copyable.
3. Open the link and check the authorization page. It must be "GitHub Copilot CLI". Previously it
   was "Theia Copilot OAuth Access" by eclipsesource, so seeing that name would mean the old flow is
   still in use.
4. Authorize, then click "I have authorized". The dialog reports success and the status bar shows the
   account.

2. Where the credentials and the sessions end up

• The OS keyring should contain a theia-copilot entry, and no new copilot-cli entries.
• No theia-copilot-login-* directory should be left behind in the temp directory.
• ~/.copilot should hold no token, and copilot on the command line should not list the prompts sent
  from Theia; those live in <Theia config dir>/copilot and are deleted per request.

3. Models and chat

• The backend log shows Copilot: using the Copilot CLI at ... and Copilot: discovered N models [...]
  with the current lineup, not just GPT-4o, and without both a family and its dated releases.
• Select a copilot/... model in the AI Configuration and send a chat message. The answer streams.

4. Tool calling

• Ask an agent something that needs a workspace tool, for example "how many first level directories does
  this workspace have?" with the Universal agent.
• The tool has to actually run and show up as a tool call in the chat, not be described as text.

5. Sign out and isolation

This is the point of the credential design, so it is worth a careful look.

• Copilot: Sign Out, confirm. The status bar shows signed out, the models become unavailable, and the
  theia-copilot keyring entry is gone.
• While signed out, an active gh auth login on the machine must not make Theia appear signed in.
  Same for GH_TOKEN or GITHUB_TOKEN set for the backend process.

6. Finding the CLI, and the upgrade path

• With ai-features.copilot.executablePath empty, the CLI is found on the PATH or in the global npm
  directory. Starting the application from a desktop launcher rather than a shell is the interesting
  case, since the npm prefix is often not on the PATH there.
• Set the preference to a path that does not exist: signing in must fail with a message naming that path.
• With a profile that was signed in using the previous version: on first start a notification explains
  that the previous sign-in was removed and asks for a new one, and the old theia-copilot-auth keyring
  entry is gone.

Optional, if available: a Copilot Business seat needs no host configuration, and for GitHub Enterprise
with data residency set ai-features.copilot.enterpriseUrl before signing in.

Follow-ups

Breaking changes

• This PR introduces breaking changes and requires careful review. If yes, the breaking changes section in the changelog has been updated.

Attribution

Builds on the work of @jdubois in #17629, which is the basis of this change.

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines
• User-facing text is internationalized using the nls service (for details, please see the Internationalization/Localization section in the Coding Guidelines)

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[richter-mvtec]

Thank you for the pullrequest!
I tried it and I can confirm, that this makes all the models expected from my copilot subscription available in theia.
I noticed, from the README it's not clear, that both @github/copilot and @github/copilot-sdk have to be installed locally.
I mentioned it also in the other PR, I'm still not sure what the impact on prompt caching is, when the chat history is merged to one long chat message.
At last, I wonder if it's necessary to use the OAuth workflow from copilot cli. From my experience, the OAuth code and client_id from the old code could be used with the Copilot SDK. This would simplify the code.

[ndoschek]

Thanks a lot for testing this, and for confirming that the models show up for you as well! 👍

1. @github/copilot-sdk: good hint, thanks. Only the CLI should be needed, since its platform package
   ships the SDK next to the executable (package/copilot and package/copilot-sdk/index.js), and that is
   the copy we load. On Linux that works, so I suspect the lookup of the copilot.cmd on Windows is
   the problem and needs adapting. I will test it in a Windows VM. The README should say this more clearly
   in any case, I will have another look and update that.

2. The sign-in: you are right that it would work and be less code. I am hesitant to hard-code an OAuth app
   client id in Theia again, and letting the CLI do the login means the app the user authorizes is the one
   actually making the requests. If the device flow with your own app is what you want for your use cases,
   we can look at making the authentication replaceable.

3. Prompt caching: your reading of the code is right, and we have no measurements yet. The SDK exposes no cache
   controls, and we start a fresh session per request and delete it afterwards, so nothing the CLI keeps per
   session survives a turn. A follow-up would be one CLI session per Theia chat session (the request carries
   a sessionId), sending only the new turn. On the agent level integration: @planger, what is your view on that?

[richter-mvtec]

About the @copilot/sdk:
I tested this inside a vscode devcontainer. Apparently copilot is preinstalled there without the sdk next to it. Also when I use the curl|bash command from the official website the cli binary is installed without the sdk.

[planger]

Prompt caching: your reading of the code is right, and we have no measurements yet. The SDK exposes no cache
controls, and we start a fresh session per request and delete it afterwards, so nothing the CLI keeps per
session survives a turn. A follow-up would be one CLI session per Theia chat session (the request carries
a sessionId), sending only the new turn. On the agent level integration: @planger, what is your view on that?

I'm not sure I understand what you mean by "agent level integration". But re prompt caching: I would assume, we break the internal prompt caching by merging and recreating a new session. I'm not very familiar with the Copilot quota accounting, but I also guess this will eat up the quota more quickly than it would need to.

So not necessarily as part of this PR but as a follow-up, it might be good to look into mapping a Theia session to a Copilot CLI session. We did something similar with the Claude Code integration. The only caveat is that when users switch language models (from something else to Copilot), there is no corresponding Copilot CLI session yet. As a fallback, we could do what we do right now (merging and creating a new session) at the cost is that they'd loose prompt caching after each switch, but I think this is acceptable.

[sdirix]

@planger I don't think Copilot has any cross-model caching, so the fallback should be fine.

[ndoschek]

@EclipseSourceAI

[EclipseSourceAI]

Note

Autonomous AI review.

This review was done by an AI agent and therefore may contain mistakes. Feel free to ignore any comment you disagree with. A thumbs-down reaction on a comment marks it as rejected for follow-up reviews. Noting why in a reply helps, since replies are read too.

Resolving all AI comments does not lead to an automatic approval. A maintainer still needs to review and sign off on the overall architecture and design.

To get an updated review after pushing changes, a maintainer may re-request a review from this account.

Running in Eclipse Enclave, submitted via review-guard-mcp

Replaces the Copilot REST transport in @theia/ai-copilot with the official Copilot CLI, spoken to over @github/copilot-sdk. The CLI is a prerequisite on the backend host (CopilotCliLocator), its bundled SDK copy is loaded at runtime (CopilotSdkLoader) against a hand-mirrored type file plus a CI drift check, the sign-in is driven via copilot login --device-code against a private temporary Copilot home whose token is taken over into Theia's keystore, and Theia's tools are bridged into the CLI session as custom tools while the agent prompt becomes the session system message. Old OAuth app, CopilotOAuthConfig, CopilotLanguageModel and the openai dependency are gone.

The approach holds up: the credential isolation (private home, useLoggedInUser: false, token variables stripped from the child env), the empty runtime mode with availableTools: ['custom:*'], the per-request session that is deleted afterwards, and the @preDestroy teardown per frontend connection all look deliberate and are backed by a solid test suite (137 tests pass, lint and compile clean here).

Where a maintainer should look:

• Tool results are handed to the CLI raw instead of through formatToolCallContentForModel, so MCP HTML results and denied-confirmation errors reach the model as JSON blobs.
• Sign-in lifecycle in CopilotCliAuthProvider: the loginResult deferred can reject with no consumer attached (unhandled rejection), cancelLogin drops it without settling it (a waitForSignIn RPC then hangs), and there is no @preDestroy, so a login in flight survives a closed connection.
• Docs: Migration.md lists a preference (ai-features.copilot.useSdk) that never existed on master, the changelog still has a #TBD link, and the README does not cover the case @richter-mvtec hit (a CLI without a bundled SDK).
• Behaviour change: ai-features.copilot.enterpriseUrl now silently has no effect while signed in.

Also worth a maintainer decision: the fragile parts by nature are the ones that parse CLI output (device code, account label, failure line) and scrape the persisted token by regex from the private home. They are well tested and well commented, but they will break on CLI changes with no compile-time signal, unlike the SDK mirror. I did not verify anything in the UI, since this needs a live Copilot subscription and an installed CLI.

[EclipseSourceAI]

Note

Autonomous AI review.

This review was done by an AI agent and therefore may contain mistakes. Feel free to ignore any comment you disagree with. A thumbs-down reaction on a comment marks it as rejected for follow-up reviews. Noting why in a reply helps, since replies are read too.

Resolving all AI comments does not lead to an automatic approval. A maintainer still needs to review and sign off on the overall architecture and design.

To get an updated review after pushing changes, a maintainer may re-request a review from this account.

Running in Eclipse Enclave, submitted via review-guard-mcp

Follow-up review after the latest push. Went through the diff since my last review (commit 3da7373 to 147ad01) and it addresses every point I raised:

• Tool results now go through formatToolCallContentForModel for the model while the structured result still reaches the stream/UI.
• CopilotCliAuthProvider: the login Deferred now has an observer attached immediately (no more unhandled rejection), cancelLogin resolves it with false before dropping the reference (no more hung waitForSignIn), and a new @preDestroy hook cancels an in-flight login when the frontend connection closes. All three are backed by the new cancelLogin unit test and the provider is bound in the same connection-scoped container as CopilotSdkClientProvider, so the lifecycle hook actually fires.
• The previously non-localized strings across CopilotCliAuthProvider, CopilotSdkClientProvider and the language models manager are now wrapped in nls.localize.
• enterpriseUrl changes while signed in now trigger a notification prompting a fresh sign-in instead of silently doing nothing.
• Docs/CI cleanup: Node 22 to 24 in the mirror workflow, the changelog #TBD link, the stale useSdk preference in Migration.md, the missing copilot-sdk-mappers export, and the leftover blanket any disable are all gone. The README now documents the CLI-without-bundled-SDK case raised by @richter-mvtec.

I checked each fix against the surrounding code (connection-scoped binding for the @preDestroy hook, the ToolHandler/formatToolCallContentForModel types, the DialogError shape used in the auth dialog) and didn't find anything newly broken. No new inline comments from me this round.

These previous comments can be resolved as they are now handled:

• raw tool results reaching the model instead of going through formatToolCallContentForModel
• loginResult deferred could reject with no consumer attached
• cancelLogin left a pending waitForLogin hanging forever
• CopilotCliAuthProvider had no @preDestroy, leaking an in-flight login on connection close
• ai-features.copilot.enterpriseUrl silently had no effect while signed in
• README missing the CLI-without-bundled-SDK case
• doc/Migration.md referencing a preference that never existed on master

I can't resolve them myself as I would need write permission on this repository.

[planger]

Excellent, tested login with and without confirming in Copilot. Handles both cases gracefully. Tested streaming and tool calling, both work fine.

Thank you very much for bringing the Copilot integration finally to a useful state!