diff --git a/skills/rasa-building-flows/SKILL.md b/skills/rasa-building-flows/SKILL.md index 682fd78..0d26f91 100644 --- a/skills/rasa-building-flows/SKILL.md +++ b/skills/rasa-building-flows/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.9.0" docs-url: https://rasa.com/docs/reference/primitives/flows/ --- @@ -30,7 +31,9 @@ metadata: (see "Branching and conditions"). 10. Extract reusable sub-tasks into child flows via `call` (see "Connecting flows"). 11. Connect follow-up flows via `link` if a separate process should start after. -12. Validate the project. +12. Ensure all referenced slots, responses, and actions exist in the domain + (see "Ensure domain completeness"). +13. Validate the project (see "Fixing validation errors"). ## Scoping a flow @@ -343,6 +346,27 @@ collect_patient_info: if: False ``` -## Full reference +## Ensure domain completeness - \ No newline at end of file +Every element a flow references must be defined in the domain (`domain.yml` or split +domain files). After writing or editing a flow, walk through each step and verify the +items below are present. Missing entries cause validation errors or silent runtime +failures. + +- Every slot used in the flow → `slots:` + (see `rasa-managing-slots`) +- Every `collect: slot_name` → `utter_ask_{slot_name}` in `responses:` + (see `rasa-writing-responses`) +- Every `action: utter_*` step → matching entry in `responses:` +- Every custom action / validation action → `actions:` + (see `rasa-writing-custom-actions`) +- Rejection responses → `responses:` + +## Fixing validation errors + +When validation fails, preserve the user's **intent** — what the flow is supposed to +accomplish. The flow steps themselves can change, but the intended behavior must not be +lost. + +Never strip down a flow's intended behavior to make validation pass. If needed, fix the +domain: slots, responses, and actions to support what the user asked for. diff --git a/skills/rasa-building-skills/SKILL.md b/skills/rasa-building-skills/SKILL.md new file mode 100644 index 0000000..20c8ffd --- /dev/null +++ b/skills/rasa-building-skills/SKILL.md @@ -0,0 +1,118 @@ +--- +name: rasa-building-skills +description: > + Builds skills (capabilities) for a Rasa CALM assistant. Use when the user asks to + create, add, or build a skill, capability, or feature for their Rasa assistant. +license: Apache-2.0 +metadata: + author: rasa + version: "0.1.0" + rasa_version: ">=3.14.0" +--- + +# Building Skills for a Rasa Assistant + +## Terminology + +In Rasa, a **skill** is a self-contained module inside a Rasa assistant that packages a +specific capability or domain (e.g. booking, payments, order tracking). + +The term "skill" can also refer to a IDE agent skill (SKILL.md). Use conversation +context to determine which meaning the user intends. If ambiguous, **ask**. + +A Rasa skill defines its interface (slots / memory) and logic (flows, prompts, actions, +or tools). It is implemented as either: + +1. **Flow-based** — a bundle of flows and slots for guided, deterministic behavior. +2. **Sub-agent** — a sub-agent that plans and acts autonomously using tools. +3. **Hybrid** — an orchestrator flow that delegates some steps to a sub-agent. + +## Workflow + +1. Clarify what the user wants the Rasa assistant to do. +2. Choose the implementation approach (see "Choosing the approach"). +3. Design the skill boundary — slots, flows, responses, actions, sub-agents + (see "Designing the skill boundary"). +4. Implement using the appropriate skill: + - Flow-based → `rasa-building-flows` + - ReAct sub-agent → `rasa-setting-up-react-agents` + - A2A sub-agent → `rasa-setting-up-a2a-agents` +5. Ensure the new flows or sub-agents are reachable (via `call`/`link` steps or + triggerable by their description). + +## Choosing the approach + +The first design choice is how autonomous the skill should be: + +``` +if high-risk domain (payments, auth, KYC, PII, compliance): + use flow-based +elif business logic must be tightly controlled (strict validation, fixed steps): + use flow-based +elif most steps are deterministic but some need autonomy: + use hybrid (flow orchestrates, sub-agent handles open-ended steps) +else: + use sub-agent (start autonomous, add flows later for guardrails as needed) +``` + +If the approach is not obvious from context, ask the user whether they want more control +over the conversation (flow-based) or a more autonomous experience (sub-agent). Frame it +in terms of the trade-off: flows give predictability and auditability, sub-agents give +flexibility and a more natural interaction. This is especially important when the user +asks for multiple skills at once — each skill may warrant a different approach, so +confirm the intent per skill before implementing. + +## Designing the skill boundary + +Before implementing, outline: + +1. **Slots** — the skill's interface. See `rasa-managing-slots`. +2. **Flows** — one per user goal. See `rasa-building-flows`. +3. **Responses** — See `rasa-writing-responses`. +4. **Actions** — backend operations. See `rasa-writing-custom-actions`. +5. **Sub-agents** — MCP servers and tools. See `rasa-setting-up-react-agents` or + `rasa-setting-up-a2a-agents`. + +### File organization + +A single-flow skill can live in one file (e.g. `data/track_order.yml`). When a skill +contains multiple flows, group them in a subdirectory named after the skill: + +``` +data/ +├── track_order.yml # single-flow skill +├── booking/ # multi-flow skill +│ ├── book_flight.yml +│ ├── select_destination.yml +│ └── choose_seats.yml +├── payments/ +│ ├── process_payment.yml +│ └── verify_payment.yml +domain/ +├── booking.yml +├── payments.yml +sub_agents/ +├── policy_search/ +│ └── config.yml +``` + +## Related skills + +| Skill | Use for | +|-------|---------| +| `rasa-building-flows` | Writing flows, branching logic, `call`/`link` steps | +| `rasa-managing-slots` | Defining slots, types, mappings, persistence | +| `rasa-writing-responses` | Bot responses and `utter_ask_*` templates | +| `rasa-writing-custom-actions` | Custom actions and validation actions | +| `rasa-setting-up-react-agents` | ReAct sub-agents with MCP tools | +| `rasa-setting-up-a2a-agents` | External sub-agents via A2A protocol | +| `rasa-calling-mcp-tools-from-flows` | Invoking MCP tools directly from flow steps | +| `rasa-configuring-mcp-server` | MCP server setup in `endpoints.yml` | +| `rasa-configuring-assistant` | Pipeline, policies, `config.yml` / `endpoints.yml` | +| `rasa-writing-e2e-tests` | End-to-end conversation tests | + + +## Examples + +For worked examples (flight booking, stock research, payments, insurance claims, Jira), +see [references/examples.md](references/examples.md). \ No newline at end of file diff --git a/skills/rasa-building-skills/references/examples.md b/skills/rasa-building-skills/references/examples.md new file mode 100644 index 0000000..74d3c07 --- /dev/null +++ b/skills/rasa-building-skills/references/examples.md @@ -0,0 +1,66 @@ +# Skill examples + +### "Create a skill that books flights" + +Well-defined multi-step process → **flow-based**. + +- **Flows**: `book_flight`, `select_destination` (child), `choose_seats` (child) +- **Slots**: `departure_city`, `destination_city`, `travel_date`, `return_date`, + `passenger_count`, `seat_preference`, `selected_flight` +- **Actions**: `action_search_flights`, `action_book_flight` + +→ `rasa-building-flows` + +### "Create a skill that explores stock options" + +Open-ended research with dynamic tool selection → **sub-agent**. + +- **Flows**: `stock_research` (thin wrapper calling the sub-agent) +- **Sub-agent**: `stock_explorer` — ReAct agent with financial data MCP server +- **Slots**: `portfolio_id` (input), `analysis_result` (output) + +→ `rasa-setting-up-react-agents` + +### "Create a payment skill" + +High-risk financial domain → **flow-based**. + +- **Flows**: `process_payment`, `select_payment_method` (child), + `verify_payment` (child) +- **Slots**: `payment_amount`, `payment_method`, `card_last_four`, `payment_confirmed` +- **Actions**: `action_validate_payment`, `action_process_payment` + +→ `rasa-building-flows` + +### "Create an insurance claim skill" + +Deterministic collection but open-ended policy search → **hybrid**. + +- **Flows**: `file_insurance_claim` (orchestrator), `confirm_and_submit_claim` (child) +- **Sub-agent**: `policy_search_agent` — searches policy clauses autonomously +- **Slots**: `claim_type`, `incident_date`, `incident_description`, `matching_policy` + +```yaml +flows: + file_insurance_claim: + description: Helps users file an insurance claim and find relevant policy coverage. + steps: + - collect: claim_type + - collect: incident_date + - collect: incident_description + - call: policy_search_agent + exit_if: + - slots.matching_policy is not null + - action: utter_claim_summary + - call: confirm_and_submit_claim +``` + +→ `rasa-building-flows` + `rasa-setting-up-react-agents` + +### "Create a Jira ticket skill" + +Depends on scope — ask the user: + +- Fixed fields, single API call → **flow-based** with a custom action. +- Needs to search projects, look up tickets, pick from many operations → + **sub-agent** with Jira MCP server. diff --git a/skills/rasa-calling-mcp-tools-from-flows/SKILL.md b/skills/rasa-calling-mcp-tools-from-flows/SKILL.md new file mode 100644 index 0000000..3ece2fe --- /dev/null +++ b/skills/rasa-calling-mcp-tools-from-flows/SKILL.md @@ -0,0 +1,131 @@ +--- +name: rasa-calling-mcp-tools-from-flows +description: > + Calls MCP tools directly from Rasa flow steps. Use when invoking an MCP tool via a + call step, mapping slot values to tool parameters, or extracting tool results into + slots. This replaces custom action code for simple API integrations. +license: Apache-2.0 +metadata: + author: rasa + version: "0.1.0" + rasa_version: ">=3.14.0" + docs-url: https://rasa.com/docs/pro/build/mcp-integration +--- + +# Calling MCP Tools from Flows + +MCP tools can be invoked directly from flow steps using a `call` step with explicit +input/output mappings. This replaces the need for custom action code when you only need +to call an external API. + +MCP servers must be defined in `endpoints.yml` before calling their tools. See the +`rasa-configuring-mcp-server` skill for server setup and authentication. + +This feature is in **beta** and available starting from **Rasa 3.14.0**. + +## When to use MCP vs a custom action + +Use an MCP tool call when: + +- The tool already exists on an MCP server +- You just need to pass slot values in and store results back +- No complex logic around the call (branching, retries, multiple sequential API calls) + +Use a custom action instead when: + +- You need complex business logic (multiple API calls, conditionals, error handling) +- You need access to the full tracker (conversation history, events, sender ID) +- You need to send custom messages via the dispatcher +- The data transformation can't be expressed in a Jinja2 output mapping +- The external API isn't exposed via MCP + +See `rasa-writing-custom-actions` for custom action guidance. + +## Workflow + +1. Ensure the MCP server is defined in `endpoints.yml` + (see `rasa-configuring-mcp-server` skill). +2. Call the MCP tool from a flow using a `call` step with `mcp_server` and `mapping` + (see "Calling an MCP tool from a flow"). +3. Map tool results to slots, handling both structured and unstructured output + (see "Tool result formats"). +4. Validate the project. + +## Calling an MCP tool from a flow + +Use a `call` step with `mcp_server` and `mapping` to invoke a tool directly. + +| Key | Required | Description | +|------------------|----------|-------------| +| `call` | yes | Tool name as exposed by the MCP server | +| `mcp_server` | yes | Must exactly match a `name` in `endpoints.yml`. Fails at runtime if mismatched | +| `mapping.input` | yes | Maps slot values to tool parameters. Each entry: `param` (tool parameter name) + `slot` (Rasa slot name) | +| `mapping.output` | yes | Maps tool results back to slots. Each entry: `slot` (target slot) + `value` (Jinja2 expression to extract the result) | + +```yaml +flows: + buy_order: + description: helps users place a buy order for a particular stock + steps: + - collect: stock_name + - collect: order_quantity + - action: check_feasibility + next: + - if: slots.order_feasible is True + then: + - call: place_buy_order + mcp_server: trade_server + mapping: + input: + - param: ticker_symbol + slot: stock_name + - param: quantity + slot: order_quantity + output: + - slot: order_status + value: result.structuredContent.order_status.success + - else: + - action: utter_invalid_order + next: END +``` + +## Tool result formats + +MCP tools return results in one of two formats. Handle both in your output mapping. + +### Structured content + +When the tool defines an output schema, Rasa returns structured data. Access specific +values using dot notation in the `value` field: + +```yaml +# Tool returns: {"result": {"structuredContent": {"order_status": {"success": true, "order_id": "abc123"}}}} +output: + - slot: order_status + value: result.structuredContent.order_status.success + - slot: order_id + value: result.structuredContent.order_status.order_id +``` + +### Unstructured content + +When the tool has no output schema, the entire result is a serialized string. Capture +it in a single slot for downstream processing: + +```yaml +# Tool returns: {"result": {"content": [{"type": "text", "text": "{\"order_status\": ...}"}]}} +output: + - slot: order_data + value: result.content +``` + +## Common pitfalls + +- **MCP server name mismatch** — the `mcp_server` value in a flow `call` step must + exactly match a `name` in `endpoints.yml`. A typo means the call fails at runtime + with no compile-time warning. +- **Missing tool on server** — if the tool name in `call` doesn't exist on the + referenced MCP server, the call fails at runtime. +- **Ignoring unstructured content** — when a tool has no output schema, results come + back as a serialized string under `result.content`, not as `result.structuredContent`. + Always account for both formats. diff --git a/skills/rasa-configuring-assistant/SKILL.md b/skills/rasa-configuring-assistant/SKILL.md index 975dc0f..7a6a383 100644 --- a/skills/rasa-configuring-assistant/SKILL.md +++ b/skills/rasa-configuring-assistant/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.13.0" docs-url: https://rasa.com/docs/pro/build/configuring-assistant --- @@ -58,10 +59,10 @@ embeddings, and input limits: pipeline: - name: CompactLLMCommandGenerator llm: - model_group: openai_llm # references model_groups in endpoints.yml + model_group: my_llm # references model_groups in endpoints.yml flow_retrieval: embeddings: - model_group: openai_embeddings # references model_groups in endpoints.yml + model_group: my_embeddings # references model_groups in endpoints.yml user_input: max_characters: 420 ``` @@ -116,15 +117,15 @@ multi-deployment routing, failover, and self-hosted models. ```yaml model_groups: - - id: openai_llm + - id: my_llm models: - - provider: openai - model: gpt-4o-2024-11-20 + - provider: # e.g. openai, azure, self-hosted + model: - - id: openai_embeddings + - id: my_embeddings models: - - provider: openai - model: text-embedding-3-large + - provider: + model: ``` ### Action endpoint @@ -159,6 +160,11 @@ nlg: The rephraser (`nlg: type: rephrase`) is covered by the `rasa-rephrasing-responses` skill. +### MCP servers + +MCP server configuration (`mcp_servers` in `endpoints.yml`) is covered by the +`rasa-configuring-mcp-server` skill. + ### Silence handling Controls how long the assistant waits before assuming the user is silent. Only applies @@ -168,7 +174,3 @@ to voice-stream channels (Twilio, Browser Audio, Genesys, Jambonz, Audiocodes). interaction_handling: global_silence_timeout: 7 # seconds, default: 7 ``` - -## Full reference - - \ No newline at end of file diff --git a/skills/rasa-configuring-mcp-server/SKILL.md b/skills/rasa-configuring-mcp-server/SKILL.md new file mode 100644 index 0000000..d60333b --- /dev/null +++ b/skills/rasa-configuring-mcp-server/SKILL.md @@ -0,0 +1,95 @@ +--- +name: rasa-configuring-mcp-server +description: > + Configures MCP (Model Context Protocol) servers in a Rasa CALM assistant. Use when + defining MCP servers in endpoints.yml, setting up authentication (API key, OAuth 2.0, + pre-issued token), or connecting to multiple external services. +license: Apache-2.0 +metadata: + author: rasa + version: "0.1.0" + rasa_version: ">=3.14.0" + docs-url: https://rasa.com/docs/reference/integrations/mcp-servers +--- + +# Configuring MCP Servers + +MCP servers expose external tools — APIs, databases, services — to a Rasa assistant. All +MCP servers are defined in `endpoints.yml` and shared across direct flow tool calls and +ReAct sub agents. + +This feature is in **beta** and available starting from **Rasa 3.14.0**. + +## Workflow + +1. Add each MCP server to `endpoints.yml` (see "Server definition"). +2. Add authentication if the server requires it (see "Authentication"). +3. Validate the project. + +## Server definition + +Register every MCP server in `endpoints.yml`. + +| Key | Required | Description | +|--------|----------|-------------| +| `name` | yes | Unique identifier — referenced from flows and sub agent configs. Duplicates cause a validation error at startup | +| `url` | yes | URL where the MCP server is running | +| `type` | yes | `http` or `https` | + +```yaml +# endpoints.yml +mcp_servers: + - name: trade_server + url: http://localhost:8080 + type: http + - name: payment_server + url: https://api.payment-service.com + type: https +``` + +## Authentication + +Add auth fields directly to the server entry in `endpoints.yml` when the MCP server +requires credentials. Sensitive values (`api_key`, `token`, `client_secret`) **must** +use `${ENV_VAR}` syntax — plain text is rejected by validation. + +### API key + +Sent as `Authorization: Bearer ` by default. Add `header_name` and +`header_format` to override the header. + +```yaml +mcp_servers: + - name: secure_api_server + url: https://api.example.com + type: https + api_key: "${API_KEY}" + header_name: "X-API-Key" # optional, default: Authorization + header_format: "{key}" # optional, default: Bearer {key} +``` + +### OAuth 2.0 (client credentials) + +```yaml +mcp_servers: + - name: oauth_server + url: https://api.example.com + type: https + oauth: + client_id: "${CLIENT_ID}" + client_secret: "${CLIENT_SECRET}" + token_url: "https://auth.example.com/oauth/token" + scope: "read:data write:data" # optional + audience: "https://api.example.com" # optional + timeout: 10 # optional +``` + +### Pre-issued token + +```yaml +mcp_servers: + - name: token_server + url: https://api.example.com + type: https + token: "${ACCESS_TOKEN}" +``` diff --git a/skills/rasa-configuring-model-groups/SKILL.md b/skills/rasa-configuring-model-groups/SKILL.md index 82d33fb..4be95c2 100644 --- a/skills/rasa-configuring-model-groups/SKILL.md +++ b/skills/rasa-configuring-model-groups/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.11.0" docs-url: https://rasa.com/docs/pro/deploy/llm-routing --- @@ -54,25 +55,25 @@ The simplest setup — one deployment per group: ```yaml model_groups: - - id: openai_llm + - id: my_llm models: - - provider: openai - model: gpt-4o-2024-11-20 + - provider: openai # or azure, self-hosted, etc. + model: - - id: openai_embeddings + - id: my_embeddings models: - - provider: openai - model: text-embedding-3-large + - provider: openai # or azure, huggingface_local, etc. + model: ``` To switch providers, change `provider` and add any required provider-specific settings: ```yaml model_groups: - - id: azure_llm + - id: my_llm models: - provider: azure - deployment: rasa-gpt-4 + deployment: api_base: https://my-azure-instance/ api_version: "2024-02-15-preview" api_key: ${AZURE_API_KEY} @@ -84,20 +85,21 @@ Place multiple deployments in the same group for load balancing, failover, or la optimization. Add a `router` block to control distribution. Keep deployments in a group on the same underlying model — mixing fundamentally -different models (e.g. GPT-3.5 vs GPT-4) leads to unpredictable behavior. Router -settings are per-group and independent — each group can use a different strategy. +different models (e.g. a small model vs a large model) leads to unpredictable +behavior. Router settings are per-group and independent — each group can use a +different strategy. ```yaml model_groups: - id: azure_llm models: - provider: azure - deployment: gpt-4-france + deployment: my-deployment-france api_base: https://azure-france/ api_version: "2024-02-15-preview" api_key: ${AZURE_KEY_FRANCE} - provider: azure - deployment: gpt-4-canada + deployment: my-deployment-canada api_base: https://azure-canada/ api_version: "2024-02-15-preview" api_key: ${AZURE_KEY_CANADA} @@ -208,10 +210,10 @@ model_groups: - id: litellm_proxy_llm models: - provider: litellm_proxy - model: gpt-4-instance-1 + model: api_base: "http://localhost:4000" - provider: litellm_proxy - model: gpt-4-instance-2 + model: api_base: "http://localhost:4000" router: routing_strategy: least-busy diff --git a/skills/rasa-managing-slots/SKILL.md b/skills/rasa-managing-slots/SKILL.md index 29acd15..ea14948 100644 --- a/skills/rasa-managing-slots/SKILL.md +++ b/skills/rasa-managing-slots/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.12.0" docs-url: https://rasa.com/docs/reference/primitives/slots/ --- @@ -183,7 +184,3 @@ different conventions, match those instead. - Prefix booleans with `is_` or `has_`: `is_verified`, `has_insurance` - Avoid special characters `(`, `)`, `=`, `,` in slot names — they break response button payloads - -## Full reference - - diff --git a/skills/rasa-rephrasing-responses/SKILL.md b/skills/rasa-rephrasing-responses/SKILL.md index 918c75e..ff25d4e 100644 --- a/skills/rasa-rephrasing-responses/SKILL.md +++ b/skills/rasa-rephrasing-responses/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.11.0" docs-url: https://rasa.com/docs/reference/primitives/contextual-response-rephraser/ --- @@ -40,13 +41,13 @@ altering meaning. nlg: type: rephrase llm: - model_group: gpt-4o-2024-11-20-openai-model # optional, defaults to gpt-4o-2024-11-20 + model_group: my_llm # optional, defaults to the default model model_groups: - - id: gpt-4o-2024-11-20-openai-model + - id: my_llm models: - - provider: openai - model: gpt-4o-2024-11-20 + - provider: # e.g. openai, azure, self-hosted + model: temperature: 0.3 # 0.0–2.0, default 0.3 ``` @@ -136,7 +137,3 @@ responses: Suggested: {{suggested_response}} Rephrased: ``` - -## Full reference - - \ No newline at end of file diff --git a/skills/rasa-setting-up-a2a-agents/SKILL.md b/skills/rasa-setting-up-a2a-agents/SKILL.md new file mode 100644 index 0000000..b646729 --- /dev/null +++ b/skills/rasa-setting-up-a2a-agents/SKILL.md @@ -0,0 +1,244 @@ +--- +name: rasa-setting-up-a2a-agents +description: > + Connects external sub agents to a Rasa CALM assistant via the A2A (Agent-to-Agent) + protocol. Use when adding an external agent, configuring an agent card, setting up + A2A authentication, customizing input/output processing, or invoking an external + sub agent from a flow. +license: Apache-2.0 +metadata: + author: rasa + version: "0.1.0" + rasa_version: ">=3.14.0" + docs-url: https://rasa.com/docs/reference/config/agents/external-sub-agents +--- + +# Connecting A2A External Sub Agents + +External sub agents connected via the A2A protocol operate as autonomous entities that +handle complex, multi-turn conversations independently. When invoked through a `call` +step in a flow, the external agent takes control of the conversation until its task is +complete. + +This feature is in **beta** and available starting from **Rasa 3.14.0**. + +## Workflow + +1. Ensure the external agent is running and exposes an A2A-compatible endpoint with an + agent card. +2. Create the sub agent directory: `sub_agents//` with a `config.yml` + (see "Directory structure"). +3. Set `agent.protocol: A2A` and point `configuration.agent_card` to the agent card + file or URL (see "Configuration"). +4. Add authentication if the external agent requires it (see "Authentication"). +5. Invoke the sub agent from a flow using an autonomous `call` step + (see "Invoking from a flow"). +6. Optionally customize input/output processing with a Python module + (see "Customization"). +7. Validate the project. + +## Directory structure + +Each A2A sub agent lives in its own subdirectory under `sub_agents/`. Both `rasa train` +and `rasa run` scan this directory by default; pass `--sub-agents ` to either +command to use a different directory. + +The agent name must be unique across all sub agents **and** all flow IDs. + +``` +your_project/ +├── config.yml +├── domain/ +├── data/flows/ +└── sub_agents/ + └── car_shopping_agent/ + ├── config.yml # required + ├── agent_card.json # local agent card (or use a URL instead) + └── custom_agent.py # optional customization module +``` + +## Configuration + +The `config.yml` requires `agent.protocol: A2A` and a `configuration.agent_card` +pointing to either a local JSON file (relative to project root) or a remote URL. + +```yaml +# sub_agents/car_shopping_agent/config.yml + +agent: + name: car_shopping_agent + protocol: A2A + description: "Helps users shop for cars by connecting them with dealers" + +configuration: + agent_card: ./sub_agents/car_shopping_agent/agent_card.json +``` + +| Key | Required | Description | +|----------------------------|----------|-------------| +| `agent.name` | yes | Unique name — must not clash with any flow ID | +| `agent.protocol` | yes | Must be `A2A` for external sub agents | +| `agent.description` | yes | Brief description of the agent's capabilities | +| `configuration.agent_card` | yes | Path or URL to the A2A agent card | +| `configuration.module` | no | Python class path for customization | + +### Agent card + +Do **not** create the agent card yourself — it is supplied by the external agent's +provider. Your job is to obtain it (as a JSON file or URL) and reference it in +`configuration.agent_card`. Rasa reads the card at startup to resolve the agent's +endpoint, transport, and auth requirements, then health-checks the connection. If the +agent is unreachable, startup fails. + +## Authentication + +Add an `auth` section under `configuration` in the sub agent's `config.yml` +(`sub_agents//config.yml`) when the external agent requires credentials. +Sensitive values (`api_key`, `token`, `client_secret`) **must** use `${ENV_VAR}` +syntax — plain text is rejected by validation. + +### API key + +Sent as `Authorization: Bearer ` by default. Add `header_name` and +`header_format` to override the header. + +```yaml +configuration: + agent_card: ./sub_agents/shopping_agent/agent_card.json + auth: + api_key: "${API_KEY}" + header_name: "X-API-Key" # optional, default: Authorization + header_format: "{key}" # optional, default: Bearer {key} +``` + +### OAuth 2.0 (client credentials) + +```yaml + auth: + oauth: + client_id: "${CLIENT_ID}" + client_secret: "${CLIENT_SECRET}" + token_url: "https://auth.company.com/oauth/token" + scope: "read:users" +``` + +### Pre-issued token + +```yaml + auth: + token: "${ACCESS_TOKEN}" +``` + +## Invoking from a flow + +Use an autonomous `call` step to delegate part of a conversation to the external +agent. The agent name must match `agent.name` in the sub agent's `config.yml`. + +```yaml +flows: + shop_for_car: + description: Helps users browse and purchase a car through an external agent. + steps: + - collect: user_budget + - call: car_shopping_agent # runs until the external agent signals completion + - action: utter_purchase_summary +``` + +Do **not** use `exit_if` — it is only supported for ReAct sub agents. The external +agent controls its own completion via the A2A protocol. + +## Task vs Message responses + +When designing the external agent's response behavior: + +- Use **Tasks** for the main workflow — they support status tracking and allow the + agent to signal `COMPLETED`. +- Use **Messages** only for clarifications — Rasa maps every `Message` to + `INPUT_REQUIRED`, so the conversation stays open and never completes. +- Never rely on Messages alone for the main operations — use a Task-only or hybrid + approach. + +## Customization + +By default Rasa sends all conversation slots to the external agent and passes the +agent's response straight back. Customization is **optional** — only create a custom +class when you need to: + +- **Filter input** — limit which slots reach the external agent (e.g. send only + `user_budget` and `car_type`, not every slot in the conversation). +- **Map output to slots** — extract structured data from the agent's response and + store it in Rasa slots so downstream flow steps can branch on it. + +### Creating a custom A2A agent class + +1. Create a Python file in the sub agent directory (e.g. + `sub_agents/car_shopping_agent/custom_agent.py`). +2. Subclass `A2AAgent` and override `process_input` and/or `process_output`. +3. Point `configuration.module` to the class. + +```python +from rasa.agents.protocol.a2a.a2a_agent import A2AAgent +from rasa.agents.schemas import AgentInput, AgentOutput +from rasa.sdk.events import SlotSet + +class CarShoppingAgent(A2AAgent): + async def process_input(self, input: AgentInput) -> AgentInput: + input.slots = [s for s in input.slots if s.name in {"user_budget", "car_type"}] + return input + + async def process_output(self, output: AgentOutput) -> AgentOutput: + if output.structured_results: + results = output.structured_results[-1] + output.events = output.events or [] + output.events.append(SlotSet("selected_car", results)) + return output +``` + +```yaml +# sub_agents/car_shopping_agent/config.yml +agent: + name: car_shopping_agent + protocol: A2A + description: "Helps users shop for cars" + +configuration: + agent_card: ./sub_agents/car_shopping_agent/agent_card.json + module: "sub_agents.car_shopping_agent.custom_agent.CarShoppingAgent" +``` + +### What you can modify + +**`process_input(input: AgentInput) -> AgentInput`** — modify what the external agent +receives. Key fields on `AgentInput`: + +| Field | Type | What to do with it | +|------------------------|------|--------------------| +| `slots` | `List[AgentInputSlot]` | Filter to only relevant slots | +| `user_message` | `str` | Rewrite or augment the user message | +| `conversation_history` | `str` | Trim or redact sensitive history | +| `metadata` | `Dict[str, Any]` | Inject custom metadata | + +**`process_output(output: AgentOutput) -> AgentOutput`** — modify what comes back into +Rasa. Key fields on `AgentOutput`: + +| Field | Type | What to do with it | +|-------|------|--------------------| +| `events` | `Optional[List[SlotSet]]` | Add `SlotSet` events to store data in Rasa slots | +| `structured_results` | `Optional[List]` | Read raw results from agent tool calls | +| `response_message` | `Optional[str]` | Rewrite the message sent to the user | + +## Common pitfalls + +- **Missing `protocol: A2A`** — without this, Rasa defaults to the RASA protocol and + treats the agent as a ReAct sub agent, which will fail. +- **Agent name clashes with flow IDs** — the agent name must be unique across all + flows and all other sub agents. +- **Agent unreachable at startup** — Rasa health-checks the agent card endpoint on + boot. If the external agent is down, Rasa will not start. +- **Using `exit_if` with A2A agents** — `exit_if` is only supported for ReAct sub + agents. A2A agents control their own completion via the A2A protocol. +- **Message-only response pattern** — responding only with `Message` objects keeps the + conversation in `INPUT_REQUIRED` indefinitely. Use `Task` objects for the main + workflow so the agent can signal `COMPLETED`. +- **User messages during processing** — messages sent while the sub agent is still + processing are not handled. Users must wait for `INPUT_REQUIRED` or `COMPLETED`. diff --git a/skills/rasa-setting-up-enterprise-search/SKILL.md b/skills/rasa-setting-up-enterprise-search/SKILL.md index d79e9e8..4edb3d6 100644 --- a/skills/rasa-setting-up-enterprise-search/SKILL.md +++ b/skills/rasa-setting-up-enterprise-search/SKILL.md @@ -8,7 +8,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.13.0" docs-url: https://rasa.com/docs/pro/build/configuring-enterprise-search --- @@ -94,13 +95,13 @@ Corresponding model groups in `endpoints.yml`: model_groups: - id: enterprise_search_llm models: - - provider: openai - model: gpt-4.1-mini-2025-04-14 + - provider: # e.g. openai, azure, self-hosted + model: - id: enterprise_search_embeddings models: - - provider: openai - model: text-embedding-3-large # default ingestion model + - provider: + model: # must match ingestion model ``` Extractive search Q&A ingestion format — `page_content` holds the question @@ -287,7 +288,3 @@ vector_store: api_key: ${SEARCH_API_KEY} collection: my_collection ``` - -## Full reference - - diff --git a/skills/rasa-setting-up-react-agents/SKILL.md b/skills/rasa-setting-up-react-agents/SKILL.md new file mode 100644 index 0000000..f2ce79d --- /dev/null +++ b/skills/rasa-setting-up-react-agents/SKILL.md @@ -0,0 +1,318 @@ +--- +name: rasa-setting-up-react-agents +description: > + Configures ReAct sub agents in a Rasa CALM assistant. Use when creating a sub-agent + that dynamically selects MCP tools, choosing between general-purpose and task-specific + agent types, customizing prompts, filtering tools, adding custom Python tools, or + overriding input/output processing. +license: Apache-2.0 +metadata: + author: rasa + version: "0.1.0" + rasa_version: ">=3.14.0" + docs-url: https://rasa.com/docs/reference/config/agents/react-sub-agents +--- + +# Configuring ReAct Sub Agents + +ReAct sub agents are built-in autonomous agents that dynamically choose which MCP +tools to invoke based on conversation context. They operate in a ReAct +(Reasoning + Acting) loop — the agent reasons about the user's request, picks a tool, +observes the result, and repeats until the task is done. + +MCP servers must be defined in `endpoints.yml` before configuring a ReAct sub agent. +See the `rasa-configuring-mcp-server` skill for server setup and authentication. + +This feature is in **beta** and available starting from **Rasa 3.14.0**. + +## Workflow + +1. Ensure the MCP server is defined in `endpoints.yml` (see `rasa-configuring-mcp-server` + skill). +2. Create the sub agent directory with a `config.yml` + (see "Directory structure" and "Configuration"). +3. Choose between general-purpose or task-specific agent type + (see "General-purpose vs task-specific"). +4. Optionally filter which MCP tools the agent can access + (see "Tool filtering"). +5. Invoke the sub agent from a flow using a `call` step + (see "Invoking from a flow"). +6. Optionally customize the prompt, input/output processing, or add custom tools + (see "Customization"). +7. Validate the project. + +## Directory structure + +Each ReAct sub agent lives in its own subdirectory under `sub_agents/`. Both +`rasa train` and `rasa run` scan this directory by default; pass `--sub-agents ` +to either command to use a different directory. + +The agent name must be unique across all sub agents **and** all flow IDs. + +``` +your_project/ +├── config.yml +├── endpoints.yml +├── domain/ +├── data/flows/ +└── sub_agents/ + └── stock_explorer/ + ├── config.yml # required + ├── prompt_template.jinja2 # optional + └── custom_agent.py # optional +``` + +## Configuration + +The sub agent's `config.yml` connects the agent to one or more MCP servers defined in +`endpoints.yml`. The protocol defaults to `RASA` — do **not** set it to `A2A`. + +```yaml +# sub_agents/stock_explorer/config.yml +agent: + name: stock_explorer + description: "Agent that helps users research and analyze stock options" + +configuration: + llm: # optional, default model is provided by Rasa codebase + model_group: my_llm + prompt_template: sub_agents/stock_explorer/prompt_template.jinja2 # optional + timeout: 30 # optional, seconds before timing out + max_retries: 3 # optional, MCP connection retries + include_date_time: true # optional, default: true + timezone: "America/New_York" # optional, default: "UTC" + +connections: + mcp_servers: + - name: trade_server + include_tools: + - find_symbol + - get_company_news + - fetch_live_price +``` + +| Key | Required | Description | +|-----|----------|-------------| +| `agent.name` | yes | Unique name — must not clash with any flow ID or other sub agent | +| `agent.description` | yes | Brief description of the agent's capabilities | +| `configuration.llm` | no | LLM to power the agent's reasoning. Has a default model | +| `configuration.prompt_template` | no | Path to a Jinja2 prompt template | +| `configuration.timeout` | no | Seconds before timing out. No timeout by default | +| `configuration.max_retries` | no | MCP connection retry attempts. Default: 3 | +| `configuration.include_date_time` | no | Include current date/time in prompts. Default: true | +| `configuration.timezone` | no | IANA timezone (e.g. `"UTC"`, `"Europe/London"`). Default: `"UTC"` | +| `configuration.module` | no | Python class path for customization | +| `connections.mcp_servers` | yes | List of MCP servers this agent connects to. At least one required | + +### Tool filtering + +For each MCP server entry under `connections.mcp_servers`, use `include_tools` or +`exclude_tools` to control which tools the agent can access. These are **mutually +exclusive** — use one or the other per server, never both. + +- `include_tools` — only these tools are available to the agent. +- `exclude_tools` — all tools except these are available. + +```yaml +connections: + mcp_servers: + - name: trade_server + include_tools: + - find_symbol + - get_company_news + - name: analytics_server + exclude_tools: + - admin_analytics +``` + +## General-purpose vs task-specific + +Rasa supports two types of ReAct sub agents. Choose based on how the agent signals +completion. + +| | General-purpose | Task-specific | +|---|---|---| +| **When to use** | Open-ended tasks where the agent decides when it's done | Structured data collection (form filling, booking) | +| **Completion** | Agent calls a built-in `task_completed` tool | Automatic when `exit_if` slot conditions are met | +| **Built-in tools** | `task_completed` only | `set_slot_` for each slot in `exit_if` | +| **Final response** | Sends a summary message to the user | Completes silently — flow continues to next step | +| **Base class** | `MCPOpenAgent` | `MCPTaskAgent` | + +## Invoking from a flow + +### General-purpose (no exit conditions) + +The agent runs autonomously until it calls `task_completed`: + +```yaml +flows: + stock_research: + description: helps research and analyze stock investment options + steps: + - call: stock_explorer +``` + +### Task-specific (with exit conditions) + +The agent runs until the specified slot conditions are met. Rasa automatically provides +`set_slot_` tools for each slot in `exit_if`: + +```yaml +flows: + appointment_booking: + description: helps users book appointments + steps: + - call: booking_agent + exit_if: + - slots.appointment_time is not null + - collect: final_confirmation +``` + +## Customization + +Customization is **optional**. Only create a custom class when you need to: + +- **Customize the prompt** — add specific instructions or pass slot values as context. +- **Filter input** — limit which slots reach the agent. +- **Map output to slots** — extract structured data from the agent's tool results. +- **Add custom Python tools** — tools that run alongside MCP tools. + +### Custom prompt template + +Create a Jinja2 file and reference it in `configuration.prompt_template`. Available +variables: + +- `{{ description }}` — the agent's description from `config.yml`. +- `{{ slots. }}` — any slot value. +- `{{ conversation_history }}` — full dialogue transcript. +- `{{ current_datetime }}` — datetime object (when `include_date_time` is enabled). + Use methods like `{{ current_datetime.strftime("%d %B, %Y") }}`. + +### Creating a custom ReAct agent class + +1. Create a Python file in the sub agent directory (e.g. + `sub_agents/stock_explorer/custom_agent.py`). +2. Subclass `MCPOpenAgent` (general-purpose) or `MCPTaskAgent` (task-specific). +3. Override `process_input`, `process_output`, and/or `get_custom_tool_definitions`. +4. Point `configuration.module` to the class. + +```python +from rasa.agents.protocol.mcp.mcp_open_agent import MCPOpenAgent +from rasa.agents.schemas import AgentInput, AgentOutput, AgentToolResult +from rasa.sdk.events import SlotSet + +class StockAnalysisAgent(MCPOpenAgent): + async def process_input(self, input: AgentInput) -> AgentInput: + input.slots = [s for s in input.slots if s.name in {"portfolio_id", "risk_level"}] + return input + + async def process_output(self, output: AgentOutput) -> AgentOutput: + if output.structured_results: + results = output.structured_results[-1] + output.events = output.events or [] + output.events.append(SlotSet("analysis_result", results)) + return output +``` + +```yaml +# sub_agents/stock_explorer/config.yml +agent: + name: stock_explorer + description: "Agent that helps users research and analyze stock options" + +configuration: + module: "sub_agents.stock_explorer.custom_agent.StockAnalysisAgent" + +connections: + mcp_servers: + - name: trade_server +``` + +### What you can modify + +**`process_input(input: AgentInput) -> AgentInput`** — modify what the agent receives. +Key fields on `AgentInput`: + +| Field | Type | What to do with it | +|-------|------|--------------------| +| `slots` | `List[AgentInputSlot]` | Filter to only relevant slots | +| `user_message` | `str` | Rewrite or augment the user message | +| `conversation_history` | `str` | Trim or redact sensitive history | +| `metadata` | `Dict[str, Any]` | Inject custom metadata | + +**`process_output(output: AgentOutput) -> AgentOutput`** — modify what comes back +into Rasa. Key fields on `AgentOutput`: + +| Field | Type | What to do with it | +|-------|------|--------------------| +| `events` | `Optional[List[SlotSet]]` | Add `SlotSet` events to store data in Rasa slots | +| `structured_results` | `Optional[List]` | Read raw results from agent tool calls | +| `response_message` | `Optional[str]` | Rewrite the message sent to the user | + +### Adding custom tools + +Implement `get_custom_tool_definitions` to add Python tools alongside MCP tools. Each +tool definition follows the OpenAI function calling spec and must include a +`tool_executor` key pointing to an async method. + +The `tool_executor` method receives `arguments` as a dict and must return an +`AgentToolResult`: + +| Field | Type | Description | +|-------|------|-------------| +| `tool_name` | `str` | Name of the tool | +| `result` | `Optional[str]` | The tool's output | +| `is_error` | `bool` | Whether the execution failed. Default: `False` | +| `error_message` | `Optional[str]` | Error details if execution failed | + +Tool executors must be async. Do **not** use blocking calls (`time.sleep`, +synchronous `requests`). Use `httpx.AsyncClient`, `asyncio.to_thread`, etc. + +```python +from typing import Any, Dict, List + +class StockAnalysisAgent(MCPOpenAgent): + def get_custom_tool_definitions(self) -> List[Dict[str, Any]]: + return [{ + "type": "function", + "function": { + "name": "recommend_stocks", + "description": "Analyze results and return stock recommendations", + "parameters": { + "type": "object", + "properties": { + "search_results": { + "type": "string", + "description": "The search results to analyze", + }, + }, + "required": ["search_results"], + "additionalProperties": False, + }, + "strict": True, + }, + "tool_executor": self._recommend_stocks, + }] + + async def _recommend_stocks(self, arguments: Dict[str, Any]) -> AgentToolResult: + results = arguments["search_results"] + return AgentToolResult(tool_name="recommend_stocks", result=results) +``` + +## Common pitfalls + +- **Agent name clashes with flow IDs** — the sub agent name must be unique across all + flows and all other sub agents. +- **Using `exit_if` with general-purpose agents** — `exit_if` is only for task-specific + agents. General-purpose agents signal completion via `task_completed`. +- **`include_tools` and `exclude_tools` on the same server** — these are mutually + exclusive per MCP server entry. Using both causes a validation error. +- **MCP server name mismatch** — the `name` under `connections.mcp_servers` must + exactly match a `name` in `endpoints.yml`. +- **Blocking calls in custom tool executors** — tool executors are async. Using + `time.sleep` or synchronous HTTP clients blocks the event loop. +- **Invalid timezone** — `configuration.timezone` must be a valid IANA timezone name. + Invalid values raise a `ValidationError` during agent initialization. +- **User messages during processing** — messages sent while the sub agent is still + processing are not handled. Users must wait for the agent to complete or reach + `INPUT_REQUIRED`. diff --git a/skills/rasa-simulating-conversations/SKILL.md b/skills/rasa-simulating-conversations/SKILL.md new file mode 100644 index 0000000..bc050e3 --- /dev/null +++ b/skills/rasa-simulating-conversations/SKILL.md @@ -0,0 +1,461 @@ +--- +name: rasa-simulating-conversations +description: > + Use when a user wants to evaluate or test a Rasa assistant using the eval scenario framework — at any stage of the process: setting up the evaluation infrastructure for the first time, deciding which scenario types to cover for a flow, writing or generating scenario YAML files, learning about supported assertion types or how to write good success criteria, running goal-driven simulations where an LLM plays the user role against a live Rasa bot, or analyzing whether existing scenarios are comprehensive. Covers both the authoring side ("write scenarios", "generate evals", "help me write criteria") and the execution side ("simulate a conversation", "run my eval scenarios"). Not for scripted end-to-end tests, unit tests, NLU training, or general Rasa bot development. +metadata: + author: rasa + version: "0.1.0" + rasa_version: ">=3.17.0" + docs-url: https://rasa.com/docs/pro/testing/simulation-evaluation/ +allowed-tools: >- + Read Glob Grep Write Edit TodoWrite ToolSearch + Bash(date:*) Bash(ls:*) Bash(cat:*) Bash(mkdir:*) Bash(find:*) Bash(grep:*) + mcp__rasa-tools +--- + +# Goal-Driven Simulation for Rasa + +Simulate realistic user conversations against a live Rasa assistant to catch +flow failures, missing slot handling, and poor bot responses — before shipping. + +--- + +## How this works + +1. You write (or generate) a **scenario YAML** in `eval/scenarios/` +2. This skill calls the `evaluate_agent` MCP tool +3. The tool uses an LLM to play the user role, driving the conversation via the REST API +4. After each run it checks **assertions** (deterministic) and **success criteria** (LLM judge) +5. Results land in `eval/results///run_N.txt` + +--- + +## Scenario YAML format + +Currently only the text-based simulation is supported. + +```yaml +scenario: + name: User successfully completes + + simulation_context: > + + + setup: + initial_slots: # injected before conversation starts + authenticated: true # set any slots that should be pre-filled + + goals: + criteria: # evaluated by LLM judge + - Agent collects all required information without confusion + - Agent confirms the task was completed successfully + - Conversation ends naturally + assertions: # deterministic checks against the tracker + - flow_started: + flow_ids: [] # one or more flow ids + operator: any # any | all + - flow_completed: + flow_id: # singular; map; optional flow_step_id + flow_step_id: # optional + - flow_cancelled: + flow_id: + - action_executed: + - slot_was_set: + - name: # set to any non-null value + - name: + value: "" # set to an exact value + - slot_was_not_set: + - name: + - bot_uttered: + utter_name: # any of utter_name / text_matches / buttons + text_matches: "regex pattern" + - bot_did_not_utter: + text_matches: "I don't know" + - sequencing: # ordered: each step matches an event strictly after the previous step's match + - flow_started: transfer_money + - slot_was_set: recipient # slot name only (string); no value match + - action_executed: action_submit_transfer + - flow_completed: # NOTE: plain string inside sequencing + # also available as steps: flow_cancelled, flow_interrupted +``` + +### Supported assertion types +Every assertion is a single-key item. These are the ONLY supported types +(validated by `validate_scenario`); anything else fails schema validation. +| Type | Example | What it checks | +|-----------------------------------|-----------------------------------------------------------------|----------------------------------------------------------------------------| +| `flow_started` | `flow_started: {flow_ids: [transfer_money], operator: any}` | one/all of these flows started | +| `flow_completed` | `flow_completed: {flow_id: transfer_money}` | flow completed (optionally at `flow_step_id`) | +| `flow_cancelled` | `flow_cancelled: {flow_id: transfer_money}` | flow was cancelled | +| `action_executed` | `action_executed: action_submit_transfer` | this action or `utter_*` ran | +| `slot_was_set` | `slot_was_set: [{name: recipient, value: "John"}]` | slot set (to `value` if given) at any point — checked over event history | +| `slot_was_not_set` | `slot_was_not_set: [{name: recipient}]` | slot was never set (to `value` if given) | +| `bot_uttered` | `bot_uttered: {text_matches: "balance"}` | bot sent a response matching `utter_name` / `text_matches` (regex) / `buttons` | +| `bot_did_not_utter` | `bot_did_not_utter: {text_matches: "I don't know"}` | bot did not send such a response | +| `generative_response_is_relevant` | `generative_response_is_relevant: {threshold: 0.7}` | LLM-judged relevance of the generated reply ≥ threshold | +| `generative_response_is_grounded` | `generative_response_is_grounded: {threshold: 0.7}` | LLM-judged grounding of the generated reply ≥ threshold | +| `pattern_clarification_contains` | `pattern_clarification_contains: [transfer_money, check_balance]` | clarification offered these flows | +| `sequencing` | see example above | the listed steps occurred in order (values are plain strings) | + +Notes: +- `flow_started` uses **plural `flow_ids` + `operator`**; `flow_completed` / + `flow_cancelled` use **singular `flow_id`** in a map. The terse + `flow_started: ` string form still works but is **deprecated**. +- `sequencing` supports exactly these six step types, each a single-key item + with a **plain string** value: `action_executed`, `slot_was_set`, + `flow_started`, `flow_completed`, `flow_cancelled`, `flow_interrupted`. + - Inside `sequencing` the value is always a bare string — even + `flow_completed`/`flow_cancelled` (top-level these need a `{flow_id: ...}` + map), and `slot_was_set` (top-level this needs a `[{name, value}]` list; + here it is just the slot name and does not check the value). + - `flow_interrupted` is available **only** inside `sequencing`. + - It checks **order only** (each step after the previous, not necessarily + adjacent), not slot/response values. + - `generative_response_is_relevant` and `pattern_clarification_contains` assertions are only relevant for single-turn knowledge-based questions. +- **Prefer structural assertions over `bot_uttered` / `bot_did_not_utter` with + `text_matches`.** Assertions should verify *business logic*, not wording — the + bot rephrases every run, so regexes over response text are both brittle (false + failures on wording drift) and weak (they don't prove the underlying behavior). + Express the intent with structural checks instead: a flow result + (`flow_completed` / `flow_cancelled`), an action (`action_executed`, including + `utter_*`), a slot (`slot_was_set` / `slot_was_not_set`), or ordering + (`sequencing`). For "the bot must not reveal X", prefer + `bot_did_not_utter: {utter_name: ...}` or assert the relevant flow never + completed, rather than a regex over the wording. + - Use `text_matches` **only when a specific literal token must appear verbatim + and that token is the business logic** — e.g. a product or subscription-plan + name, an account/reference number, a URL, or a code. Do not use it to check + generic phrasing like "not found", "sorry", or "your bill is". +--- + +## Step-by-step workflow + +### Step 1 — Check eval/conftest.yml + +Before doing anything else, check whether `eval/conftest.yml` exists: + +``` +Use Glob("eval/conftest.yml") to check. +``` + +**If it exists** — read it and confirm the required fields are present (see below). Proceed to Step 2. + +**If it does not exist** — create it with the exact content below, **including every comment line** — do not omit or paraphrase them. Then tell the user to verify or fill in the details before continuing: + +```yaml +# eval/conftest.yml — simulation configuration + +# LLM used to drive the user simulator (generates user turns) +simulation: + llm: + provider: openai # openai | anthropic | azure | ... + model: gpt-5.1 + timeout: 30 + reasoning_effort: "none" + cache: + no-cache: true + +# LLM used to evaluate success criteria (LLM judge) +evaluation: + llm: + provider: openai + model: gpt-5.1 + timeout: 30 + reasoning_effort: "none" + cache: + no-cache: true + # Prompt overrides — paths are relative to the project root. + # Use these to customise how the judge scores conversations. + # prompt_template: prompt_templates/judge.jinja2 # overrides text/criteria judge (variables: simulation_context, criteria_text, transcript) + +``` + +Stop and tell the user: + +``` +eval/conftest.yml has been created. Fill in the provider and model details, +then say "continue" and I'll pick up from here. +``` + +Do not proceed until the user confirms. + +### Step 2 — Ensure the bot is trained and running + +Before simulating, verify the assistant is live: + +``` +Use talk_to_assistant with message ["hello"] to verify the bot responds. +If it fails, ask the user to run: + rasa run --enable-api --inspect --cors "*" --credentials credentials.yml +``` + +The `--inspect` flag loads the Inspector in the same process, so a single server is all that's needed — no separate +`rasa inspect` command. + +**After retraining:** always check `agent_reloaded` in the `train_rasa_assistant` response. + +- If `agent_reloaded: true` — the server picked up the new model, proceed to simulate. +- If `agent_reloaded: false` — the server is not running. Do NOT attempt workarounds (e.g. SlotSet in custom actions). + Tell the user: + ``` + The model was trained but the server is not running. Start it with: + rasa run --enable-api --inspect --cors "*" + Then say "re-run the simulation" and I'll continue from here. + ``` + Stop and wait — do not re-run the simulation until the user confirms the server is up. + +### Step 3 - Ensure MCP server is running + +Verify that you have access to the following tools: `list_project_flow_definitions`, `validate_scenario` and `evaluate_agent`. +If one of the tools is not avaiable, report which tool is missing to the user and stop: + ``` + The following required MCP tools are not avaiable: (list unavaiable tools here) + Please make sure that Rasa MCP tools are installed and running. Run the setup wizard from your project root: `rasa tools init`. See the following page for more details: https://rasa.com/docs/pro/installation/rasa-mcp-tools/ + ``` + +### Step 4 — Find or create scenario files + +Scenarios live in a flat directory: + +``` +Use Glob("eval/scenarios/*.yml") to list available scenarios. +``` + +Generate scenario(s) from the flow definition: + +1. Read the flow with `list_project_flow_definitions`, then open the flow file. +2. Identify the flow's own structure: the goal it serves, the information it + collects (each slot + its description/validation), every branch/condition, + and every decision or exit point. +3. Write each scenario to `eval/scenarios/.yml`, named + after the situation (e.g. `transfer_wrong_account_corrected.yml`). + +Cover the **happy path first**, then add scenarios for realistic ways a real +user might deviate. Derive the deviations from *this* flow rather than a fixed +list — each collected input, branch, and exit point is a place where a real +conversation can diverge. Use these as thinking prompts, not a checklist to fill +mechanically: + +- The user gives information that is missing, invalid, or out of range — and may + then correct it. +- The user is indirect or vague and reveals their need gradually. +- The user changes their mind, hesitates, or wants to stop partway. +- The user goes off-topic, asks something unrelated, or asks for a human. +- The user asks for something this flow (or the bot) does not handle. +- A meaningful branch/condition in the flow is exercised (take each path). + +Aim for a handful of high-value scenarios per flow — enough to exercise the +happy path and the most likely real deviations — not one of every possible kind. +At a minimum, include: the happy path, one where the user supplies a bad value +(and corrects it), and one where the user changes their mind or stops partway. + +**Writing `simulation_context`:** write a short, natural briefing of *who the +user is and what they want*, grounded in this flow and the plausible data they'd +have. Give the simulator a persona, a goal, and the facts it needs (e.g. which +credential to use, which detail to get wrong) — then let its behavior emerge. Do +**not** script turn-by-turn what the user should say, and vary the persona +(cooperative / impatient / uncertain, terse / chatty) across scenarios so the +set isn't formulaic. The simulator only ever plays the *user* — never describe +what the bot should do here (that belongs in `criteria` / `assertions`). + +**Backend-dependent values:** some facts the simulator provides must match the +bot's connected backend / mock data for even the happy path to succeed — e.g. a +valid password or PIN, an existing order number, a known account or phone number. +Do **not** invent these blindly. If a scenario is meant to *succeed* and depends +on a value you cannot confirm is valid test data, ask the user for it before +running (e.g. *"What's a valid test order number / username / password I should +use?"*). For deviation scenarios that are *supposed* to fail (wrong password, +unknown order), inventing a clearly-invalid value is correct — there the "not +found" response is the expected behavior. + +### Writing success criteria — bot-side deal-breakers + +**Division of labor — keep these strictly separate:** + +- **`criteria`** capture **bot-side performance**: the *non-negotiables* of an + acceptable conversation. Rule of thumb: **if a criterion fails, a fix is + definitely required.** They are judged independently of whether the user + ultimately got their answer. +- **`task_completion`** is a separate **binary** metric (`true`/`false`, scored + automatically by the metrics judge — you do **not** author it here). It + captures the **actual user outcome**: did the user get what they came for? +- Do not restate the outcome as a criterion. "The user got their answer / was + satisfied / the issue was resolved" is `task_completion`, **not** a criterion. + +**Good criteria (deal-breakers) target one of these:** + +- **Step ordering / preconditions** — e.g. + `The agent verifies the customer's identity before disclosing any billing details`. +- **Correct behind-the-scenes tool calls** — e.g. + `The agent calls the billing-lookup tool before quoting any figures` + (assert it deterministically when you can — see below). +- **Flow adherence** — the required handling of a path, e.g. + `After three failed password attempts the agent stops the verification attempts and tells the customer it cannot proceed`. + (Whether it *then offers a callback* is an optional nicety — leave it out.) +- **Mandatory phrasing** — only where wording is genuinely required + (legal / compliance / safety disclosures), e.g. + `The agent states the call may be recorded before collecting personal data`. + This is the *only* case where a specific phrase is the point. + +**Do NOT write criteria for (these flicker or belong elsewhere):** + +- **User outcome / satisfaction / resolution** → that's `task_completion`. This + includes *"the agent provides/answers the billing information the user asked + for"* — receiving the answer is the user outcome, not a bot deal-breaker. +- **Offering alternatives / callbacks / escalation as a fallback** — "offers a + callback", "offers an alternative after failure". These are UX choices about + the user's resolution path, not non-negotiable behavior. (A *mandatory* + compliance escalation is the rare exception — phrase it as the required + behavior, not as "offers ...".) +- **Optional niceties** — closing pleasantries, "offers further help", upsells. + If the bot can skip it and still be correct, it is not a criterion. +- **Exact wording that isn't mandated** — the bot rephrases every run, so a + literal-phrase criterion flips when wording drifts. Describe the behavior, + not the sentence (except mandatory phrasing above). +- **Conversation-closure gates** — "ends naturally", "conversation ends": where + the simulated user stops varies run-to-run, so these flicker. + +**Two more rules:** + +- **One non-negotiable per criterion.** Compound criteria ("calls the tool + **and** explains clearly") fail ambiguously — split them. +- **Put anything deterministic in `assertions`, not `criteria`.** A tool call, a + slot set, a completed flow, step ordering — assert these against the tracker; + assertions don't flicker. Reserve `criteria` for non-negotiable behavior that + still needs an LLM to judge (e.g. "explained *why* it could not proceed"). + +> Rule of thumb: every criterion must be a **deal-breaker** — if it fails, the +> bot needs fixing. If a criterion could fail while the bot did everything right +> (wording drift, where the conversation stopped, an optional nicety), or if it +> is really about whether the *user* succeeded, it does not belong in +> `criteria`. + +### Step 5 — Validate generated scenario files + +For each scenario file that you created use the `validate_scenario` tool to validate a scenario YAML file for syntax and +assertion correctness. If validation fails, fix the scenario YAML according to the error message and re-run the scenario +validation. You need to have a valid scenario files before proceeding to the next step and running the simulation. + +### Step 6 — Run the simulation + +Before the first call, generate a shared experiment ID by running this Bash command so all scenarios from this session land in the same results folder: + +```bash +date +%Y-%m-%d_%H-%M-%S +``` +Use the output as the `experiment_id` for all `evaluate_agent` calls. If you encounter a permission error, use the currentDate from the system context. + +Run scenarios sequentially using `evaluate_agent`, passing the same `experiment_id` +to every call. + +``` +evaluate_agent( + scenario_path="eval/scenarios/.yml", + experiment_id=experiment_id +) +``` + +### Step 7 — Report results + +After the tool returns, summarize clearly: + +**If all runs passed:** + +``` +3/3 runs passed for "" +Naturalness: 8.2/10 — responses were warm and clear. +Result file: eval/results///run_N.txt +``` + +**If some runs failed:** + +``` +1/3 runs passed for "" + +Run 2 failed: + ✗ slot_filled: + slot '' = None + +Run 3 failed: + ✗ success_criteria: "Agent confirms the task was completed successfully" + Bot did not send a confirmation message before ending the conversation. + +Naturalness: 6.1/10 — functional but robotic tone. +Result file: eval/results///run_N.txt +``` + +Do not attempt to fix failures unless the user asks. Report what failed and where, then wait. + +**Before reporting a failure as a bot problem, rule out invalid mock data.** +Some failures are caused by the data the simulator supplied not existing in the +bot's backend — not by a bot defect. Tell-tale signs in the transcript: + +- backend lookup failures that depend on the supplied value — *"order not + found"*, *"no account with that number"*, *"customer not found"*; +- authentication failing on a scenario that was meant to authenticate + successfully (*"that password is incorrect"* when the run expected success). + +If the failing value came from your `simulation_context` (or `initial_slots`) and +the scenario was meant to **succeed**, this is almost certainly a **test-data +problem, not a bot bug**. In that case: + +1. Do **not** report it as a bot failure and do **not** edit the bot / flow. +2. Pause and ask the user for valid mock values, naming exactly what you need and + which scenario it's for, e.g.: + ``` + The happy-path run for "" failed because the order number I + used (12345) wasn't found in the backend. What's a valid test order number / + username / password I should use? I'll update the scenario and re-run. + ``` +3. Once the user provides them, update the affected `simulation_context` (and any + `initial_slots` / assertions that hard-code the value), then re-run that + scenario with the same `experiment_id`. + +Only scenarios that are *designed* to fail (wrong password, unknown order) keep +invalid values — there, assert the "not found" / failure behavior instead of +asking for new data. + +### Step 8 — Fix on request only + +If the user asks to fix a failure: + +1. Identify the root cause: wrong response template? missing slot handling? flow routing issue? +2. Point to the specific file and line +3. Propose the fix +4. After the fix, re-run the failing scenario to confirm it passes + +--- + +## Config reference + +| Field | Required | Default | +|----------------------------------|------------|------------------------------| +| `simulation.llm` | Yes | — | +| `evaluation.llm` | Yes | — | +| WebSocket URL | No | Derived from Rasa server URL | +| Sample rate, turn timing | No | Built-in defaults | + +--- + +## Common issues + +| Symptom | Likely cause | Fix | +|--------------------------------------------|----------------------------------------------|------------------------------------------------------------------------------------| +| `Scenario file not found` | Wrong path or missing file | Check with Glob using the right pattern (see Step 2), create file if needed | +| Bot returns empty responses (text) | Rasa not running or not trained | Ask user to run `rasa run --enable-api --inspect --credentials credentials.yml` | +| `response_contains_any` fails unexpectedly | Bot uses different phrasing | Read the transcript in the result file, update assertions or flow response | +| Simulation hits max turns (20) | Bot stuck in loop or user simulator confused | Read transcript, check if bot keeps re-asking same question | +| All criteria pass but assertions fail | Slot not being filled | Check tracker slots in result file, verify slot mappings in domain | +| Happy-path run fails with "not found" / auth errors | Mock value (order no., account, password) isn't in the backend | Test-data problem, not a bot bug — ask the user for valid test data, update `simulation_context`/`initial_slots`, re-run | + +--- + +## Viewing a run in the Inspector + +After a simulation run, the result file includes an Inspector URL under `--- INSPECTOR URL ---`. +Open it directly in the browser — the conversation is already in the shared tracker store because the server was started +with `--inspect`. + +No extra tool call or file loading needed. diff --git a/skills/rasa-writing-custom-actions/SKILL.md b/skills/rasa-writing-custom-actions/SKILL.md index 430d401..a98db33 100644 --- a/skills/rasa-writing-custom-actions/SKILL.md +++ b/skills/rasa-writing-custom-actions/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.7.0" docs-url: https://rasa.com/docs/reference/integrations/action-server/actions --- @@ -23,6 +24,10 @@ conversation proceeds (branching, conditions, collect steps). Actions do the *ra flows to use. Keep each action focused on one responsibility — `action_search_products` searches, `action_place_order` places an order. +If all you need is a simple API call with slot-based input/output mappings and the API +is available on an MCP server, consider using an MCP tool call instead — see +`rasa-calling-mcp-tools-from-flows`. + ## Workflow 1. Identify what raw work the flow needs — API call, database query, computation. @@ -261,7 +266,3 @@ actions: - action_ask_delivery_option - validate_phone_number ``` - -## Full reference - - diff --git a/skills/rasa-writing-e2e-tests/SKILL.md b/skills/rasa-writing-e2e-tests/SKILL.md index 624c3c7..5f16415 100644 --- a/skills/rasa-writing-e2e-tests/SKILL.md +++ b/skills/rasa-writing-e2e-tests/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.12.0" docs-url: https://rasa.com/docs/pro/testing/evaluating-assistant --- @@ -160,17 +161,17 @@ Specify `utter_source` to target a specific component. ground_truth: "Items can be returned within 30 days of purchase." ``` -The LLM Judge model is configured in `conftest.yml` at the project root (defaults to -OpenAI `gpt-4.1-mini`): +The LLM Judge model is configured in `conftest.yml` at the project root (if not +specified, Rasa falls back to its default model): ```yaml llm_judge: llm: - provider: openai - model: "gpt-4.1-mini-2025-04-14" + provider: # e.g. openai, azure, self-hosted + model: embeddings: - provider: openai - model: "text-embedding-3-large" + provider: + model: ``` ## Fixtures @@ -258,7 +259,3 @@ If custom actions are not stubbed, start the action server first: rasa run actions & rasa test e2e ``` - -## Full reference - - diff --git a/skills/rasa-writing-responses/SKILL.md b/skills/rasa-writing-responses/SKILL.md index 9cf4368..82f5381 100644 --- a/skills/rasa-writing-responses/SKILL.md +++ b/skills/rasa-writing-responses/SKILL.md @@ -7,7 +7,8 @@ description: > license: Apache-2.0 metadata: author: rasa - version: "3.x" + version: "0.1.0" + rasa_version: ">=3.7.0" docs-url: https://rasa.com/docs/reference/primitives/responses/ --- @@ -241,7 +242,3 @@ responses: - text: "What would you like to do today?" allow_interruptions: true # default ``` - -## Full reference - - \ No newline at end of file