diff --git a/docs.json b/docs.json index b8f20585..eed70a05 100644 --- a/docs.json +++ b/docs.json @@ -164,7 +164,8 @@ "guides/ai-agents/ai-writeback", "guides/ai-agents/content-tools", "guides/ai-agents/mcp-servers", - "guides/ai-agents/autopilot" + "guides/ai-agents/autopilot", + "guides/ai-agents/deep-research" ] }, "references/integrations/lightdash-mcp", diff --git a/guides/ai-agents/deep-research.mdx b/guides/ai-agents/deep-research.mdx new file mode 100644 index 00000000..6cac57b0 --- /dev/null +++ b/guides/ai-agents/deep-research.mdx @@ -0,0 +1,209 @@ +--- +title: "Deep Research" +sidebarTitle: "Deep Research" +description: "Kick off long-running, background AI investigations against a project's semantic layer via the Deep Research API." +--- + + + **Availability:** Deep Research is an **Enterprise** feature and is currently behind a feature flag. [Contact us](mailto:support@lightdash.com) to enable it for your workspace. + + +Deep Research runs a durable, background AI investigation against a Lightdash project and produces a structured report when it's done. Unlike an interactive AI agent chat, a Deep Research run is asynchronous — you start it with an API call, poll for its status, optionally stream progress events, and read the final report once the run reaches a terminal state. + +Deep Research is exposed as a public API today. A future release will surface the same capability inside the Lightdash UI. + +## What it does + +Deep Research uses a managed Claude agent that has read-only access to the target project. Each run: + +- Selects the project you scoped it to and discovers explores and fields through the same MCP tools an interactive agent uses. +- Runs bounded warehouse queries against your semantic layer to gather evidence. +- Synthesises a **markdown report** — an intro that answers the question, one section per finding with a confidence tag and at most one embedded chart, an optional caveats section, and a conclusion. Sources are listed at the bottom when the agent uses `[n]` citations. +- Attaches **chart snapshots** to the report so evidence renders instantly. Each chart is either a warehouse chart (a verified query from the run) or an inline chart (agent-computed data from derived analysis, capped at 100 columns × 10 rows). Warehouse charts can be refreshed on demand to fetch live data. + +The Deep Research agent is **read-only**. It has no content-write or writeback tools — it cannot create, edit, or delete charts, dashboards, or any other Lightdash content. + +**Best for:** questions that need broader exploration than a single agent turn — e.g. "Investigate why paid-search-driven revenue dropped last month across our top five product categories." + +## When to use it + +| Use Deep Research when… | Use an interactive AI agent when… | +|---|---| +| The question needs multiple queries, cross-checks, and a written report | The question is a single ask you want an answer to right now | +| You want a reusable, structured artefact (findings, evidence, caveats) | You want a conversational back-and-forth | +| You're driving it from an external system, script, or scheduled job | You're a business user in the Lightdash UI or Slack | + +## Effort tiers + +Each run has a server-owned budget determined by an **effort** tier. The budget caps runtime, model tokens, tool calls, warehouse queries, and result rows. + +| Effort | Max runtime | Max tokens | Max tool calls | Max warehouse queries | Max result rows | +|---|---|---|---|---|---| +| `low` | 15 min | 500,000 | 50 | 10 | 5,000 | +| `medium` *(default)* | 30 min | 1,000,000 | 125 | 25 | 10,000 | +| `high` | 45 min | 2,000,000 | 250 | 50 | 25,000 | +| `xhigh` | 55 min | 4,000,000 | 500 | 100 | 50,000 | + +Pick the smallest effort that reliably answers the kinds of questions you're sending. The `xhigh` runtime is capped below the 60-minute worker limit to leave the agent time to clean up gracefully. + +A run reaches a terminal state as soon as it completes, fails, is cancelled, or hits any budget cap. Terminal statuses include `completed`, `partially_completed`, `failed`, and `cancelled`. + +## Permissions + +Starting a Deep Research run requires the `create:AiDeepResearch` project scope. This scope is: + +- Part of the **AI** scope group and is **Enterprise-only**. +- Included in the **Developer** system role and inherited by **Admin**. +- **Not** granted to Editor, Interactive Viewer, or Viewer roles — those users cannot start a run even if they can otherwise view the project. +- Not automatically added to custom roles — grant the scope explicitly on any custom role that should be able to start runs. + +Creating a run also preflights the caller's ability to create and delete personal access tokens (used for the run's temporary credential) and requires `view:Project`. Reading a run, listing its events, and cancelling it are creator-owned and only require `view:Project`. + +Demo mode cannot create Deep Research runs. + +## Credential lifecycle + +Every run mints its own short-lived personal access token immediately before execution. That PAT is stored only in the managed-agent's credential vault and is deleted on any terminal state — including failure, cancellation, timeout, and budget exhaustion. **You don't create or manage this PAT** and it doesn't appear in your account's PAT list for long. + +## API reference + +All endpoints are rooted at `/api/v1/ee/projects/{projectUuid}/ai-deep-research`. Authenticate with either a session or a personal access token, the same way you'd call any other Lightdash API. + +For request and response shapes, see the auto-generated [API reference](/api-reference/v1/introduction) and search for `AiDeepResearch`. + +### Start a run + +```http +POST /api/v1/ee/projects/{projectUuid}/ai-deep-research +``` + +Body: + +```json +{ + "prompt": "Investigate why revenue from our summer sandals collection slowed down in the last 4 weeks. Break it down by acquisition channel and country.", + "effort": "medium" +} +``` + +- `prompt` *(required)* — the question or investigation brief. +- `effort` *(optional)* — one of `low`, `medium`, `high`, `xhigh`. Defaults to `medium`. +- `threadUuid` *(optional)* — link the run to an existing AI agent thread in the same project. The thread must exist, be in the same project, and be owned by the caller. +- `promptUuid` *(optional)* — link the run to a specific prompt inside that thread. Requires `threadUuid`. + +Returns `202 Accepted` with the newly created run, including `aiDeepResearchRunUuid` and the resolved `budget` snapshot. The run then executes in the background. + +### Read a run + +```http +GET /api/v1/ee/projects/{projectUuid}/ai-deep-research/{aiDeepResearchRunUuid} +``` + +Returns the durable run — its `status`, the resolved `budget`, timestamps, `prompt`, any linked `aiThreadUuid`/`promptUuid`, and the `resultMarkdown` report once the run is in a terminal state. Poll this endpoint to know when to fetch the final report. + +### List runs for a thread + +```http +GET /api/v1/ee/projects/{projectUuid}/ai-deep-research?threadUuid= +``` + +Returns the caller's Deep Research runs linked to the given AI agent thread, scoped by organisation, project, and creator. Useful for rendering run history against a chat thread from your own UI. + +### List progress events + +```http +GET /api/v1/ee/projects/{projectUuid}/ai-deep-research/{aiDeepResearchRunUuid}/events?cursor= +``` + +Returns persisted progress events in chronological order, plus a `nextCursor`. Events include status transitions, cancellation requests, and phase/activity progress (`planning`, `investigating`, `validating`, `synthesizing`). + +Pass the returned `nextCursor` back unchanged on the next call to fetch only newer events. Cursors round-trip PostgreSQL microsecond precision, so re-polling the tail after the run has already ended doesn't replay the final event. + +### Cancel a run + +```http +POST /api/v1/ee/projects/{projectUuid}/ai-deep-research/{aiDeepResearchRunUuid}/cancel +``` + +Requests cancellation of an in-flight run. The run transitions to `cancelled` on the next safe checkpoint and its temporary PAT is deleted as part of the terminal cleanup. + +### Refresh a warehouse chart + +```http +POST /api/v1/ee/projects/{projectUuid}/ai-deep-research/{aiDeepResearchRunUuid}/charts/{chartKey}/refresh +``` + +Re-executes the stored query for a warehouse chart in the report and returns fresh results. Use this to view live data instead of the snapshot the agent captured at publish time. Inline (agent-computed) charts have no underlying query and return a `400`. + +## Report shape + +The final report is a single markdown document plus a map of chart snapshots. The markdown always follows this outline: + +```markdown +Intro prose that answers the question and states overall confidence. + +## First finding +Optional caveat + +Setup prose that frames the finding. + +[Revenue by channel, last 4 weeks](#chart-channel_revenue_4w) + +Interpretation of the chart above. + +## Second finding +… + +## Caveats + +Optional list of caveats that apply to the whole report. + +## Conclusion + +- Bullet takeaway 1 +- Bullet takeaway 2 + +## Sources + +Only present when the report uses `[n]` citations. +``` + +Each `[Title](#chart-)` link is a placeholder for a chart snapshot. When rendering the report yourself, look up `` in the run's chart-data map to get the snapshot rows and chart config, then render inline. Warehouse charts also expose a live-refresh endpoint (see above). + +## Example: start, poll, and read the report + +The snippet below starts a run and polls for its status until it's terminal, then prints the markdown report. Replace `LIGHTDASH_HOST`, `LIGHTDASH_PAT`, and `PROJECT_UUID` with your own values. + +```bash +#!/usr/bin/env bash +set -euo pipefail + +BASE="$LIGHTDASH_HOST/api/v1/ee/projects/$PROJECT_UUID/ai-deep-research" +AUTH="Authorization: ApiKey $LIGHTDASH_PAT" + +# 1. Start the run +RUN_UUID=$(curl -sS -X POST "$BASE" \ + -H "$AUTH" -H 'Content-Type: application/json' \ + -d '{ + "prompt": "Investigate why revenue from our summer sandals collection slowed down in the last 4 weeks. Break it down by acquisition channel and country.", + "effort": "medium" + }' | jq -r '.results.aiDeepResearchRunUuid') + +echo "Started run $RUN_UUID" + +# 2. Poll until the run reaches a terminal state +while :; do + RUN=$(curl -sS "$BASE/$RUN_UUID" -H "$AUTH") + STATUS=$(echo "$RUN" | jq -r '.results.status') + echo "status=$STATUS" + case "$STATUS" in + completed|partially_completed|failed|cancelled) break ;; + esac + sleep 10 +done + +# 3. Print the markdown report (null on failed/cancelled runs) +echo "$RUN" | jq -r '.results.resultMarkdown' +``` + +To follow progress in real time instead of only polling the run, call `.../events?cursor=` in a loop and keep passing the latest `nextCursor` back in. diff --git a/guides/ai-overview.mdx b/guides/ai-overview.mdx index 2b8ebfc8..cc0fcff3 100644 --- a/guides/ai-overview.mdx +++ b/guides/ai-overview.mdx @@ -12,6 +12,7 @@ Lightdash provides several ways to leverage AI across your analytics workflow. E | Capability | Use case | Requirements | |------------|----------|--------------| | [Lightdash AI agents](#lightdash-ai-agents) | Answer questions and create charts using your semantic layer | Cloud Pro or Enterprise add-on | +| [Deep Research](/guides/ai-agents/deep-research) | Run long, structured background investigations against a project via API | Enterprise | | [Lightdash Data MCP](#lightdash-data-mcp) | Query your semantic layer from external LLMs | All Lightdash Cloud and Enterprise users | | [Lightdash Docs MCP](#lightdash-docs-mcp) | Give your LLM context about Lightdash features | Free for everyone | | [Agent skills for your CLI](#agent-skills) | Build and maintain your semantic layer with external AI coding agents | Free for everyone | @@ -152,6 +153,7 @@ Not sure which surface to point a user at? See [Choosing the right AI workflow i | If you want to... | Use | |-------------------|-----| | Let business users ask questions in natural language | [Lightdash AI agents](/guides/ai-agents) | +| Run a long, asynchronous investigation and get back a structured report | [Deep Research](/guides/ai-agents/deep-research) | | Query Lightdash data from Claude, ChatGPT, or custom agents | [Lightdash Data MCP](/references/integrations/lightdash-mcp) | | Help your coding AI understand Lightdash documentation | [Lightdash Docs MCP](#lightdash-docs-mcp) | | Get AI help writing YAML configurations | [Agent skills for your CLI](/guides/developer/agent-skills) |