docs: add docs/riteway-ai.md with temperature/model configuration for OpenCode

docs/riteway-ai.md

@@ -0,0 +1,179 @@
+# `riteway ai` — AI Prompt Evaluations
+
+The `riteway ai` CLI runs your AI agent prompt evaluations against a configurable pass-rate threshold. Write a `.sudo` test file, run it through any supported AI agent, and get a TAP-formatted report with per-assertion pass rates across multiple runs.
+
+---
+
+## Authentication
+
+All agents use OAuth authentication — no API keys needed. Authenticate once before running evals:
+
+| Agent | Command | Docs |
+|-------|---------|------|
+| Claude | `claude setup-token` | [Claude Code docs](https://docs.anthropic.com/en/docs/claude-code) |
+| Cursor | `agent login` | [Cursor docs](https://docs.cursor.com/context/rules-for-ai) |
+| OpenCode | See docs | [opencode.ai/docs/cli](https://opencode.ai/docs/cli/) |
+
+---
+
+## Writing a test file
+
+AI evals are written in `.sudo` files using [SudoLang](https://github.com/paralleldrive/sudolang) syntax:
+
+```
+# my-feature-test.sudo
+
+import 'path/to/spec.mdc'
+
+userPrompt = """
+Implement the sum function as described.
+"""
+
+- Given the spec, should name the function sum
+- Given the spec, should accept two parameters named a and b
+- Given the spec, should return the correct sum of the two parameters
+```
+
+Each `- Given ..., should ...` line becomes an independently judged assertion. The agent is asked to respond to the `userPrompt` (with any imported spec as context), and a judge agent scores each assertion across all runs.
+
+---
+
+## Running an eval
+
+```shell
+riteway ai path/to/my-feature-test.sudo
+```
+
+By default this runs **4 passes**, requires **75% pass rate**, uses the **claude** agent, runs up to **4 tests concurrently**, and allows **300 seconds** per agent call.
+
+```shell
+# Specify runs, threshold, and agent
+riteway ai path/to/test.sudo --runs 10 --threshold 80 --agent opencode
+
+# Use a Cursor agent with color output
+riteway ai path/to/test.sudo --agent cursor --color
+
+# Use a custom agent config file (mutually exclusive with --agent)
+riteway ai path/to/test.sudo --agent-config ./my-agent.json
+```
+
+---
+
+## Options
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--runs N` | `4` | Number of passes per assertion |
+| `--threshold P` | `75` | Required pass percentage (0–100) |
+| `--timeout MS` | `300000` | Per-agent-call timeout in milliseconds |
+| `--agent NAME` | `claude` | Agent: `claude`, `opencode`, `cursor`, or a custom name from `riteway.agent-config.json` |
+| `--agent-config FILE` | — | Path to a flat single-agent JSON config `{"command","args","outputFormat"}` — mutually exclusive with `--agent` |
+| `--concurrency N` | `4` | Max concurrent test executions |
+| `--color` | off | Enable ANSI color output |
+| `--save-responses` | off | Save raw agent responses and judge details to a companion `.responses.md` file |
+
+Results are written as a TAP markdown file under `ai-evals/` in the project root.
+
+---
+
+## Saving raw responses for debugging
+
+When `--save-responses` is passed, a companion `.responses.md` file is written alongside the `.tap.md` output. It contains the raw result agent response and per-run judge details (passed, actual, expected, score) for every assertion — useful for debugging failures without adding console noise.
+
+```shell
+riteway ai path/to/test.sudo --save-responses
+```
+
+Each test file produces its own uniquely-named pair of files (e.g. `2026-03-17-test-abc12.tap.md` and `2026-03-17-test-abc12.responses.md`), so multiple test files never conflict.
+
+### Capturing responses as CI artifacts
+
+In GitHub Actions, use `--save-responses` and upload the `ai-evals/` directory as an artifact:
+
+```yaml
+- name: Run AI prompt evaluations
+  run: npx riteway ai path/to/test.sudo --save-responses
+
+- name: Upload AI eval responses
+  if: always()
+  uses: actions/upload-artifact@v4
+  with:
+    name: ai-eval-responses
+    path: ai-evals/*.responses.md
+    retention-days: 14
+```
+
+The `if: always()` ensures responses are uploaded even when assertions fail, so you can inspect exactly what the agent produced.
+
+### Partial results on timeout
+
+If some runs complete before another times out, the completed runs' responses are still written to the responses file. The timed-out run's partial agent output is also captured, followed by a `[RITEWAY TIMEOUT]` marker showing when and where the timeout occurred. This lets you debug why a run took too long and potentially optimize the prompt to run faster.
+
+---
+
+## Custom agent configuration
+
+`riteway ai init` writes all built-in agent configs to `riteway.agent-config.json` in your project root, so you can add custom agents or tweak existing flags:
+
+```shell
+riteway ai init           # create riteway.agent-config.json
+riteway ai init --force   # overwrite existing file
+```
+
+The generated file is a keyed registry. Add a custom agent entry and use it with `--agent`:
+
+```json
+{
+  "claude":   { "command": "claude",   "args": ["-p", "--output-format", "json", "--no-session-persistence"], "outputFormat": "json"  },
+  "opencode": { "command": "opencode", "args": ["run", "--format", "json"],                                   "outputFormat": "ndjson" },
+  "cursor":   { "command": "agent",    "args": ["--print", "--output-format", "json"],                        "outputFormat": "json"  },
+  "my-agent": { "command": "my-tool",  "args": ["--json"],                                                    "outputFormat": "json"  }
+}
+```
+
+```shell
+riteway ai path/to/test.sudo --agent my-agent
+```
+
+Once `riteway.agent-config.json` exists, any agent key defined in it supersedes the library's built-in defaults for that agent.
+
+---
+
+## Configuring Model and Temperature (OpenCode)
+
+Most agent harnesses don't expose temperature settings. If you need temperature control for Riteway AI evals, use OpenCode with a custom agent config.
+
+### 1. Create the agent file
+
+Create `.opencode/agents/riteway.md` in your project root:
+
+```markdown
+---
+description: Riteway AI eval agent
+mode: primary
+model: anthropic/claude-sonnet-4-6
+temperature: 0.2
+---
+```
+
+### 2. Configure the Riteway agent
+
+In your Riteway config (`riteway.agent-config.json`), reference the agent by filename (without extension):
+
+```json
+{
+  "opencode": {
+    "command": "opencode",
+    "args": ["run", "--agent", "riteway", "--format", "json"],
+    "outputFormat": "ndjson"
+  }
+}
+```
+
+Then run your evals using the `opencode` agent:
+
+```shell
+riteway ai path/to/test.sudo --agent opencode
+```
+
+This gives you precise control over both the model and sampling temperature used during evaluations, which is useful for experiments that require deterministic or creative outputs.