pi adds documentation-lift evals for model and provider customization

christianklotz merged pull request #9491 into the pi coding agent on September 11, 2026, extending the eval suite in packages/evals with documentation-lift evals: comparative runs that quantify how much the Pi documentation embedded in the system prompt actually helps. Three customization tasks are measured — adding a model to an existing provider, adding an OpenAI-compatible provider, and adding a custom streaming provider — and the existing extension-authoring eval moves onto the same baseline. None of it is in a published release yet; it rides main until whatever follows v0.85.1 is cut.

Each comparative eval runs the same task twice: once with pi’s default system prompt, and once against a baseline that removes only the embedded Pi documentation block, via a new shared excludePiDocumentation helper keyed on the prompt’s stable section markers. The extension eval already compared prompts, but its old baseline stripped the Guidelines section along with the docs; the refactor narrows the difference to documentation alone, so the reported lift is attributable to the docs.

The two new eval files put the scenarios on real verification rather than string checks. models.eval.ts asks the agent to author a fixture model entry for an existing provider and judges the resulting runtime: the model must resolve with the expected metadata, and every pre-existing model has to survive the configuration reload. providers.eval.ts spins up a local HTTP server standing in for an OpenAI-compatible endpoint and a hand-rolled streaming API, then requires the agent’s configuration to actually round-trip — probe prompts have to come back with the expected markers before a run can score.

Repetition and reporting are first-class. --repetitions on the command line or PI_EVAL_REPETITIONS in the environment reruns the comparative suites — the README’s guidance is one pass while developing an eval and five when reporting lift — and every invocation ends with a compound Eval Comparisons report: pass-rate delta in percentage points, plus token, latency, and estimated-cost figures for both sides. Artifacts persist under .eval/ as report.txt, report.json, runs.jsonl, session transcripts in sessions/, and source attachments in sources/. The harness also isolates each run’s eval home and agent state in a temporary directory while preserving runtime reload behavior and session artifacts.

The eval commands, the report format with illustrative values, and the artifact layout are all spelled out in the pull request’s changes to packages/evals/README.md on GitHub.