Problem statement
An Assistant in Home Assistant is currently a thin group of settings: a language, a conversation agent entity, and speech-to-text/text-to-speech settings. Everything that actually determines behaviour (when powered by an LLM), the system prompt and what the agent is allowed to reach, lives one level down, on the conversation entity itself.
That was adequate when an Assistant could only turn lights on and off. It is not adequate for what Assistants are becoming. As they gain real capability, documentation retrieval (#156), management APIs, awareness of what the maintainer is looking at (#157), and persistent conversation history (#159), two problems surface.
First, behaviour is not reusable. Because the prompt and the access settings are attached to the conversation entity, a maintainer who wants two differently behaving assistants on the same model has to set up the same integration twice. There is no way to say "same brain, different job".
Second, and more importantly, a maintainer has no way to express something as basic as "this assistant may control my home, that one may also read documentation and inspect my configuration". The controls exist, but at the wrong level and in a place most maintainers never open.
This matters most for the maintainers who are least comfortable with AI. Trust in a capable assistant comes from being able to see and limit what it can do, where you configure it. Without that, the only two options are to grant everything or use nothing. Shipping more capability without shipping control makes that choice worse with every release.
Community signals
- The feature request to allow complete disabling of any AI features (discussion #2238) is the blunt version of this need. Users are asking for an off switch because there is no dial.
- The popularity of external tooling such as ha-mcp (2600+ stars) shows maintainers assembling their own assistants with their own prompts and their own chosen tool sets, because Home Assistant does not let them do it inside Home Assistant.
- Community discussion around custom prompts for conversation agents is persistent, with maintainers sharing prompt snippets and workarounds rather than using a supported field. TODO: link the two clearest examples .
- The reactions to the related AI roadmap opportunities, where "what can it actually see and do" is the recurring first question.
We are not aware of a feature request phrased as "let me configure my agent". As with context awareness, this is hard to ask for before the capability exists. The signals here are indirect but consistent: maintainers want capability and they want it bounded.
Scope & Boundaries
In scope
- Renaming Assistants to Agents throughout the UI and its supporting copy
- Lifting the system prompt and API access from the conversation entity up to the agent level as overrides, so the same
conversation entity can be reused across multiple agents
- Per-agent selection of which API sets an agent may access, distinguishing home control from home management
Not in scope
- Removing these settings from the
conversation entity. This would be a breaking change that we are not willing to make (open question: are we sure about this?)
- Tool-level or integration-level permissions within an API set. This opportunity works at the granularity of the API set only
- Entity and device exposure, which is configured separately and stays as it is for now
- Restricting agents per Home Assistant user or per device
- Building the management capabilities themselves
Foreseen solution
We rename Assistants to Agents, and the agent becomes the thing a maintainer configures, picks, and talks to. The conversation entity becomes the engine it runs on.
The reasoning behind that naming:
- We currently offer three main (voice) assistants: Alexa, Google Assistant, and Assist. Calling the configurable groups of settings inside Assist "Assistants" as well makes the word carry two meanings at once.
- Users already configure Assist through these groups, picking a language, a basic instruction, and a conversation agent entity to determine how processing is done.
- The industry has landed on a shared mental model. The Assist "brand" is our equivalent of Google Assistant, Alexa, ChatGPT, or Claude. Within ChatGPT or Claude users expect a model picker. Instead of picking just the model, we allow the user to configure the entire Agent the user interacts with.
With the rename in place, the settings that determine behaviour move up to where the maintainer already is. The system prompt and the API access settings become agent-level settings. Model-specific settings stay on the conversation entity, because they belong to the model rather than to the job the agent is doing.
Access is expressed as one toggle per API set. On launch that is home control and home management, matching the split introduced in the management APIs opportunity. Each toggle is described in plain language in terms of what the agent will be able to do, not in terms of APIs. New agents get home control enabled and home management off, so a fresh agent is immediately useful without granting anything a maintainer did not ask for.
The system prompt ships with a default that produces good behaviour out of the box. It is clearly labelled as the default, and a maintainer who edits it can restore it in one action, so experimenting with the prompt is never a one-way door.
The result is that one conversation entity can back several agents: a voice agent for the family that only controls the home, and a separate agent for the maintainer that can also help with configuration and documentation. Same model, different job, configured in one place.
Risks & open questions
- Do we support inheritance? Or do we accept a breaking change? The gentle path is to leave the prompt and access settings on the conversation entity as inherited defaults, with the agent able to override them. Nobody's setup breaks. The cost is that every behaviour question then has two possible answers, and the UI has to make the inherited-versus-overridden distinction legible in a way that non-technical maintainers actually follow. That is hard to design well and easy to design badly, and getting it wrong trades one confusing setup for another. The alternative is to take the breaking change: these settings live at the agent level, full stop, and maintainers move their values once. It is a worse upgrade day and a much clearer product. Current thinking leans towards the breaking change, but this needs a decision before execution.
Appetite
Medium, 1 cycle. The configuration UI is contained if the settings section already exists. The real investment is the override model at the agent level, the rename, and getting the maintainer-facing wording right.
Execution issues
No response
Decision log
Problem statement
An Assistant in Home Assistant is currently a thin group of settings: a language, a conversation agent entity, and speech-to-text/text-to-speech settings. Everything that actually determines behaviour (when powered by an LLM), the system prompt and what the agent is allowed to reach, lives one level down, on the
conversationentity itself.That was adequate when an Assistant could only turn lights on and off. It is not adequate for what Assistants are becoming. As they gain real capability, documentation retrieval (#156), management APIs, awareness of what the maintainer is looking at (#157), and persistent conversation history (#159), two problems surface.
First, behaviour is not reusable. Because the prompt and the access settings are attached to the
conversationentity, a maintainer who wants two differently behaving assistants on the same model has to set up the same integration twice. There is no way to say "same brain, different job".Second, and more importantly, a maintainer has no way to express something as basic as "this assistant may control my home, that one may also read documentation and inspect my configuration". The controls exist, but at the wrong level and in a place most maintainers never open.
This matters most for the maintainers who are least comfortable with AI. Trust in a capable assistant comes from being able to see and limit what it can do, where you configure it. Without that, the only two options are to grant everything or use nothing. Shipping more capability without shipping control makes that choice worse with every release.
Community signals
We are not aware of a feature request phrased as "let me configure my agent". As with context awareness, this is hard to ask for before the capability exists. The signals here are indirect but consistent: maintainers want capability and they want it bounded.
Scope & Boundaries
In scope
conversationentity can be reused across multiple agentsNot in scope
conversationentity. This would be a breaking change that we are not willing to make (open question: are we sure about this?)Foreseen solution
We rename Assistants to Agents, and the agent becomes the thing a maintainer configures, picks, and talks to. The c
onversationentity becomes the engine it runs on.The reasoning behind that naming:
With the rename in place, the settings that determine behaviour move up to where the maintainer already is. The system prompt and the API access settings become agent-level settings. Model-specific settings stay on the conversation entity, because they belong to the model rather than to the job the agent is doing.
Access is expressed as one toggle per API set. On launch that is home control and home management, matching the split introduced in the management APIs opportunity. Each toggle is described in plain language in terms of what the agent will be able to do, not in terms of APIs. New agents get home control enabled and home management off, so a fresh agent is immediately useful without granting anything a maintainer did not ask for.
The system prompt ships with a default that produces good behaviour out of the box. It is clearly labelled as the default, and a maintainer who edits it can restore it in one action, so experimenting with the prompt is never a one-way door.
The result is that one
conversationentity can back several agents: a voice agent for the family that only controls the home, and a separate agent for the maintainer that can also help with configuration and documentation. Same model, different job, configured in one place.Risks & open questions
Appetite
Medium, 1 cycle. The configuration UI is contained if the settings section already exists. The real investment is the override model at the agent level, the rename, and getting the maintainer-facing wording right.
Execution issues
No response
Decision log