Local Evals

[gabeweisz]

• Modify the repeatability test script to be able to run against a local Pi server
• Create a script to run a local server (usually vllm but sglang should work), install and configure Pi, and run local evals using it

[Copilot]

Pull request overview

This PR adds support for running the repeatability analysis/eval harness against a local “pi” agent backend, and introduces a Docker-based launcher to spin up a local OpenAI-compatible inference server, configure pi from /v1/models, and run the evals end-to-end.

Changes:

• Extend run_repeatability_parallel.sh to support --pi / USE_PI=1, unify agent invocation, and improve TEST_IDS filtering + reporting.
• Add run_pi_analysis_in_docker.sh to launch an inference server in Docker, install/configure pi + TraceLens, and run the repeatability harness using pi.
• Update generate_ref.sh TEST_IDS filtering to support underscore-prefix matching and warn when no IDs match.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
agent_evals/Analysis/eval_scripts/run_repeatability_parallel.sh	Adds pi backend support, refactors agent invocation, improves test filtering/logging, and normalizes trace/reference paths.
agent_evals/Analysis/eval_scripts/run_pi_analysis_in_docker.sh	New Docker launcher script to start a local inference server, configure pi, install TraceLens, and run evals.
agent_evals/Analysis/eval_scripts/generate_ref.sh	Improves TEST_IDS filtering semantics and adds a “no matches” warning.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Ahmedhasssan-aig]

LGTM!