Conversation Intelligence Platform

This project implements a production-oriented Conversation Intelligence Platform that performs semantic conversation understanding exactly once, projects the results into structured artifacts, detects operational anomalies, builds a semantic knowledge base, and provides a grounded Retrieval-Augmented Generation (RAG) Resolution Assistant.

The project emphasizes engineering architecture, deterministic processing, observability, and evaluation rather than prompt engineering alone.

The final assignment submission is documented in WRITEUP.md.

Quick Start

make install
make start

Open:

• http://localhost:8000

The repository includes a pre-generated snapshot of AI-derived artifacts so reviewers can immediately explore the completed platform. The full pipeline remains available and can be regenerated locally at any time.

make start brings up PostgreSQL + pgvector (waiting until healthy) and the Adminer database UI, then serves the API in the foreground and prints the URLs. Everything is discoverable from the landing page.

Repository Guide

• README.md — getting started
• WRITEUP.md — final assignment write-up
• ARCHITECTURE.md — complete architecture
• docs/ARCHITECTURE_DECISIONS.md — architectural decision records
• docs/PROMPT_LIBRARY.md — production prompts
• docs/IMPLEMENTATION_PLAN.md — implementation roadmap

Status: Phase 7 (Evaluation & Observability) complete. Every pipeline stage is independently runnable from the Control Center, CLI, or REST API. See docs/IMPLEMENTATION_PLAN.md for the full roadmap and status.

Prerequisites

• uv (Python packaging; installs Python 3.12 for you)
• Docker + Docker Compose (for PostgreSQL + pgvector)
• make

Demo Workflows

Reviewers have two supported workflows:

1. Explore the pre-generated AI dataset and immediately inspect the completed platform.
2. Run the full pipeline from source data to regenerate every AI artifact.

The canonical workflow generates every artifact locally:

make start
# In the Control Center:
# 1. Run Data Ingestion
# 2. Run Conversation Understanding
# 3. Run Remaining Pipeline

For demonstrations and evaluation, reviewers can instead restore a supplied snapshot:

make start
# In the Control Center:
# 1. Run Data Ingestion
# 2. Click "Import Pre-generated AI Dataset"

The import does not overwrite raw imported conversations or messages.

By default the Control Center imports data/processed/data-artifacts.tgz; override this with DERIVED_DATA_PATH. The same restore is available from the CLI:

app import-derived data/processed/data-artifacts.tgz

Semantic search and the Resolution Assistant may still need one embedding request per user query. If GOOGLE_API_KEY is unavailable, those query-time AI features remain disabled gracefully while imported anomaly and knowledge-base artifacts stay inspectable.

If Google AI Studio has not yet been configured, the landing page detects the missing GOOGLE_API_KEY and shows an Enable AI Capabilities card: create a free key at Google AI Studio, paste it into the card, and click Save Configuration — the key is written to your local .env and AI capabilities enable immediately, no restart or manual file editing required. The database, Adminer, API, and documentation remain fully usable without the key; only AI-powered capabilities are disabled until it is configured.

Optional: copy .env.example to .env to override any setting.

Port already in use? If the API port is already occupied, make start will show the listener and ask whether to stop it before restarting the API. Set API_PORT=8001 if you prefer to serve the API on a different port. The database publishes host port 5432 and Adminer 8080 by default. Override either: DB_HOST_PORT=5433 ADMINER_HOST_PORT=8081 make start (and set DATABASE_URL=postgresql+psycopg://cx:cx@localhost:5433/cx in .env if you moved the DB port).

Adminer auto-logs into the local dev database. Set ADMINER_AUTOLOGIN=0 in .env if you want the normal login form instead.

Control center

The landing page at http://localhost:8000 is the operational control center for the platform — the pipeline is run from here, not just observed:

Pipeline Control Center:

Anomaly Dashboard:

Resolution Assistant:

• Service Status — green/yellow/red health for PostgreSQL, pgvector, and the API.
• Pipeline stage cards — one card per stage (Data Ingestion, Conversation Understanding, Anomaly Detection, Knowledge Base, Resolution Assistant) showing its status, prerequisites, outputs, and last execution, with a Run / Run Again / Open action. Stages whose prerequisites are unmet are disabled and explain why; stages from future phases are disabled with their planned phase. Conversation Understanding exposes two explicit actions — Run Sample (100) for development and prompt iteration, and Run Full Dataset (~10k LLM calls, resumable) — plus Import Pre-generated AI Dataset for restoring a cached snapshot of derived artifacts — so a full run is always a deliberate choice. Conversation Understanding defaults to UNDERSTAND_CONCURRENCY=32; on a free-tier Gemini key, set UNDERSTAND_CONCURRENCY=1 and expect rate-limited pacing (the provider honours the server's suggested retry delay); a full run realistically needs a paid tier.
• Run Remaining Pipeline — one click executes every incomplete stage in dependency order, skipping completed stages (completion is derived from the data, so nothing reruns unnecessarily) and stopping cleanly at the first blocked or not-yet-implemented stage.
• Recent Runs — the pipeline audit trail: every stage execution is durably recorded (stage, trigger source, timing, outcome — including failures) and the latest runs are shown here. Also available via GET /api/pipeline/runs and app runs.
• AI onboarding — if GOOGLE_API_KEY is missing, the Enable AI Capabilities card accepts a key via a password-style input and saves it to your local .env (never echoed back); AI-stage prerequisites flip to met immediately.
• Anomaly Analysis panel — an operational dashboard embedded (always visible) in the Anomaly Detection stage card, which spans the full width: an hourly timeline of the top four anomalies (real conversation timestamps, so the operator sees when a spike began and how it evolved) with the selected anomaly highlighted — click a charted anomaly card to switch; the highest-severity anomaly is selected by default — one card per anomalous issue, aggregating its observation periods (worst severity, triggering signals, absolute counts first — baseline → current per day — with percent change shown only when the baseline is large enough to be meaningful, recommended action), and the raw markdown report as a collapsible section. The report remains served at GET /api/anomalies/report.
• Resolution Assistant panel — grounded decision support for agents: describe a new ticket (structured via Prompt #1, never persisted) or point at an analyzed conversation (with a per-issue picker for multi-issue conversations), and get a recommendation grounded exclusively in retrieved KnowledgeDocuments — with the cited evidence highlighted, the retrieval provenance shown, and an honest "no sufficiently similar historical resolutions were found" when the evidence isn't there. The stage card's Open button jumps here.
• Quick Actions / Links — jump to the API docs, the database UI, and the in-page Anomaly Analysis panel.

The REST API exposes the same orchestration layer used by the CLI and Control Center.

Path	Purpose
/	Control-center landing page
/docs	Swagger UI
/health	Machine health probe (JSON)
/api/status	Services, AI, stage cards, job state, metrics (backs the landing page)
POST /api/pipeline/run	Run every incomplete pipeline stage in dependency order
POST /api/pipeline/{stage}/run	Run a single pipeline stage in the background
POST /api/pipeline/import-derived	Import the configured pre-generated AI artifact snapshot
GET /api/pipeline/runs	Pipeline audit trail — recent stage runs, newest first
GET /api/anomalies	Detected anomalies (canonical artifact: signals, metrics, actions)
GET /api/anomalies/report	The anomaly report, rendered from persisted anomalies (markdown)
GET /api/anomalies/timeline	Hourly issue-frequency timeline for one anomaly issue (issue; backs the chart)
GET /api/knowledge/search	Metadata-first semantic search over the knowledge base (q, product, limit)
POST /api/resolution	Grounded resolution recommendation for one issue (conversation_id or free-text text)
GET /api/resolution/issues	The selectable issues of one analyzed conversation (backs the issue picker)
GET /api/evaluation/latest	Headline numbers of the most recent golden-dataset evaluation (backs the AI Quality panel)
GET /api/evaluation/report	The latest evaluation report, rendered from the stored run (markdown)
/api/config	Non-secret configuration (secrets reported only as set/unset)
POST /api/config/google-key	Save the Google AI Studio key from the onboarding card
http://localhost:8080	Adminer database UI (auto-login to dev database cx)

Lower-level targets

make start/stop are built on smaller targets you can also run directly: make up/down (containers only), make serve (API only), make db-health. Run make data-artifacts to refresh the local derived-data bundle at data/processed/data-artifacts.tgz.

Architecture at a Glance

For the full design, see ARCHITECTURE.md.

Project layout

src/cxintel/
  config.py          # typed settings (pydantic-settings)
  logging.py         # logging setup
  db.py              # engine, session factory, health check
  cli.py             # `app` CLI (Typer)
  api/app.py         # FastAPI app (landing page + /health, /api/status, /api/config)
  api/status.py      # typed platform-status model (backs the control center)
  api/static/        # control-center landing page
  ingestion/         # Phase 2  — dataset loading + idempotent import
  llm.py             # provider abstraction (Google AI Studio, native structured output)
  understanding/     # Phase 3  — StructuredConversation schema, Prompt #1, service
  anomaly/           # Phase 4  — deterministic detector, Prompt #3, service, report
  knowledge_base/    # Phase 5  — KnowledgeDocument, knowledge_text, embeddings, retrieval
  resolution_assistant/ # Phase 6  — ContextBundle, Prompt #2, grounded resolution service
tests/               # foundation + API smoke tests
docker/Dockerfile    # pgvector image + baked-in init scripts
docker/initdb/       # pgvector init script (runs on DB first boot)
data/raw/            # place sample_tickets_v6.json here (git-ignored)

Make targets

Target	Description
start / stop	Start the full stack (DB + Adminer + API) / stop the containers
install	Create the venv and install all dependencies (uv sync)
lock	Update the uv lockfile
up / down	Start / stop PostgreSQL + pgvector
db-reset	Recreate the database (drops the volume)
db-health	Check DB connectivity + pgvector
db-migrate	Apply database migrations
fmt	Format with Ruff
lint / lint-fix	Lint (and auto-fix) with Ruff
typecheck	Type-check with mypy (strict)
test	Run pytest
check	lint + typecheck + test (CI gate)
serve	Run the FastAPI service
clean	Remove caches and build artifacts
data-artifacts	Refresh data/processed/data-artifacts.tgz with derived AI artifacts only
ingest / stats / pipeline	Import the dataset / show ingestion stats / run remaining stages
understand	Run conversation understanding on a sample of 100 (see app understand --full)
bottlenecks	Show slow per-conversation LLM observations from understanding runs
anomaly-observations	Show slow anomaly-detection stage observations
analyze	Run deterministic anomaly detection vs the Day-1 baseline
build-kb	Build the retrieval knowledge base (deterministic docs + embeddings)
chat	Resolution Assistant — grounded recommendations from historical evidence

CLI (app)

app --help
app version
app db health | app db upgrade
app serve
app ingest         # import the dataset (idempotent; applies migrations first)
app understand     # conversation understanding — sample of 100 (resumable)
app understand --full  # process every remaining conversation (~10k LLM calls)
app import-derived data/processed/data-artifacts.tgz  # restore cached AI artifacts
app export-derived data/processed/derived-ai-dataset.zip  # write derived snapshot zip
app stats          # ingestion statistics — verifies the import
app analyze        # deterministic anomaly detection vs the Day-1 baseline
app report         # print the anomaly report (from persisted anomalies)
app build-kb       # build the knowledge base (resumable; re-embeds only changes)
app search "pod leaking water" --product "Pod 5"  # semantic knowledge search
app chat "water pooling under the hub" --product "Pod 4"  # resolve a new ticket
app chat -c conv_72912dd7 --issue 1  # resolve one issue of an analyzed conversation
app chat           # interactive mode: describe tickets, get grounded recommendations
app pipeline       # run every incomplete pipeline stage in dependency order
app runs           # pipeline audit trail — recent stage runs, newest first
app bottlenecks --sort llm_seconds  # slowest LLM observations by phase timing
app anomaly-observations --sort started_at  # anomaly detection stage timings
app evaluate       # run the golden-dataset evaluation → reports/evaluation-report.{json,md}
app evaluate --check              # validate the golden dataset only (no DB, no LLM)
app evaluate --suite retrieval    # run one suite: understanding | retrieval | resolution
app evaluate --promote-baseline   # promote the current report to the committed baseline
app evaluate --strict             # exit non-zero on any failed case or regression (CI)

Evaluation (Phase 7)

app evaluate runs the version-controlled golden dataset (evals/golden/) through the production AI code paths and compares the structured artifacts deterministically — no LLM judges (ADR-015). The report records prompt and model versions, per-suite pass rates, retrieval metrics (recall/precision@k, MRR, filter-relaxed rate), grounding metrics (citation validity, grounded accuracy, downgrades), token usage, and regressions against the committed baseline (evals/baseline/evaluation-baseline.json). Every run is persisted to the evaluation_runs table and surfaced on the Control Center's Evaluation card. See evals/golden/README.md for the case-authoring workflow. Evaluation runs only explicitly — "Run Remaining Pipeline" never triggers it.

Configuration

All settings come from environment variables (or .env); see .env.example. Key ones: GOOGLE_API_KEY, LLM_PROVIDER, LLM_MODEL, EMBEDDING_PROVIDER, EMBEDDING_MODEL, DATABASE_URL, RAW_DATA_PATH, DERIVED_DATA_PATH, SLACK_WEBHOOK_URL, LOG_LEVEL.

Pre-generated AI dataset format

DERIVED_DATA_PATH points to data/processed/data-artifacts.tgz by default. That bundle includes a derived-ai-dataset.zip file with this layout:

manifest.json
tables/conversation_analyses.csv
tables/conversation_issues.csv
tables/issue_catalog.csv
tables/anomalies.csv
tables/knowledge_documents.csv

manifest.json must use format cxintel-derived-v1 and include each table's path and row count:

{
  "format": "cxintel-derived-v1",
  "tables": {
    "conversation_analyses": { "path": "tables/conversation_analyses.csv", "rows": 10000 },
    "conversation_issues": { "path": "tables/conversation_issues.csv", "rows": 12400 },
    "issue_catalog": { "path": "tables/issue_catalog.csv", "rows": 80 },
    "anomalies": { "path": "tables/anomalies.csv", "rows": 12 },
    "knowledge_documents": { "path": "tables/knowledge_documents.csv", "rows": 6700 }
  }
}

The importer validates headers, manifest counts, and conversation references, then replaces only the derived artifact tables in one PostgreSQL transaction. If validation or import fails, the existing derived data is rolled back intact.

To refresh the reviewer artifact bundle from the current derived AI tables:

make data-artifacts

This writes data/processed/data-artifacts.tgz, which contains only derived-ai-dataset.zip plus checksum/restore notes. It does not include the raw ticket dataset or a full PostgreSQL dump.

Import it with:

app import-derived data/processed/data-artifacts.tgz

Design Philosophy

This project intentionally favors:

• deterministic processing over opaque AI workflows
• strongly typed AI artifacts
• simple operational architecture
• reproducible evaluation
• clear separation between semantic reasoning and conventional software engineering