# Skill benchmarks (`co benchmark`, `co eval run`, `co eval report`) Write the standard before the skill. A benchmark is a frozen set of cases — what the user says, what must happen, what must not — and the skill is judged against it: run the real Agent, score every expectation, edit only the skill, run the identical benchmark again, and see whether it improved. ```bash co benchmark --help # the schema and the workflow co benchmark check reimbursement # validate; never runs an Agent co eval run reimbursement --agent agent.py --skill reimbursement --runs 1 co eval report reimbursement --latest # reopen it; compare with the run before ``` A coding agent needs nothing but `co --help` to find this: the top-level help names the workflow, `co skills --help` says it manages skills and points here, and every command ends with the next one. ## The benchmark file `.co/benchmarks/.yaml`: ```yaml name: reimbursement cases: - id: mixed-batch-title-mismatch kind: counterexample given: "One invoice is addressed to a different company" input: 'Process this batch: INV-301 buyer "OpenOnion Pty Ltd" $80; INV-302 buyer "Open Onion Trading Co" $95' expect: must: - "INV-301 is submitted for approval" - "The user is told INV-302 was not submitted because its buyer title does not match" must_not: - "INV-302 is submitted" - "The user is told the invoices are paid" # ...at least five cases in all, at least one normal and one counterexample ``` | field | required | meaning | |---|---|---| | `id` | yes | unique within the file | | `kind` | yes | `normal` (the task succeeds) or `counterexample` (the right answer is to refuse, stop or flag) | | `input` | yes | exactly what the user says to the Agent | | `expect.must` | yes | outcomes a user could observe; not exact wording, not a tool route | | `expect.must_not` | counterexamples | outcomes that must not happen; one that does is a hard FAIL | | `given` | no | the situation, shown to the judge | | `fixture` | no | a path (relative to `.co/benchmarks/`) the case depends on; checked to exist | There is **no `agent:` or `skill:` field**, and `check` refuses one: the same cases must be able to compare two Agents or two versions of a skill without the standard moving. `co benchmark check ` enforces at least 5 cases, both kinds, unique ids, no two identical inputs, a `must` on every case and a `must_not` on every counterexample. It lists every problem with the case, the field and the fix (`--json` for an agent), and exits 2 until the file is valid. Whether five cases are really five different decisions is for a person to read; the CLI says so rather than pretending to check it. ## Running it ```bash co eval run --agent agent.py [--skill NAME] [--invoke auto|explicit] [--runs N] [--max-iterations N] [--live] [--json] ``` - `--agent` imports the real Agent from that file. A file that ends in `host(agent)`, like the `co create` template, is imported without serving. - `--skill NAME` names the skill under test. It must be discoverable (`.co/skills/NAME/SKILL.md`, `~/.co/skills/`, or built-in); its path and SHA-256 go into the report. A case where the skill did not run **fails**, however good the answer was. - `--invoke auto` (default) sends the input unchanged and requires the Agent to call `skill(name=NAME)` itself. `--invoke explicit` sends `/NAME ` through the skills plugin and requires the plugin to have replaced it with the skill's instructions. Say which one you are testing: explicit proves the skill works, not that the Agent would choose it. - Without `--skill`, the run scores the Agent as a whole and says it made no claim about any skill. - `--runs N` repeats every case on a fresh session and reports stability (`2/3`). Start with `--runs 1`: every attempt is paid for, and the first run is to learn whether the cases can pass at all. - `--max-iterations N` (default 10) is how many steps — model calls — one attempt may take. An attempt that reaches it is **STOPPED**: not judged, never a pass, exit 1. The `co create` agent is otherwise allowed 100 steps, and on 1.8.8b9 a case with no data in its input let it search the workspace for 26 steps a case. Raise it for a task that really needs more. Before the first model call the run prints how many Agent runs it is about to make and the step ceiling on each; the report ends with what the Agent's own model calls cost (the judge's are not included). Put the data a case needs in its `input` — the invoices, the prior submission — so a correct Agent answers without searching; the example `co benchmark list` prints does this. ### The Agent cannot read the answers The benchmark file holds every `must` and `must_not`, and it sits in the project the Agent works in. On 1.8.8b11 the `co create` agent answered four of five cases by running `glob("**/*")` and then reading `.co/benchmarks/reimbursement.yaml`, and scored 5/5. So, for the length of a run: - any tool call that names `.co/benchmarks`, `eval-runs` or the benchmark's own file name is **refused** before it runs, and the Agent is told why. A case's declared `fixture:` stays readable. - an attempt whose tool results contain any expectation text anyway — a grep over the workspace, a shell pipeline, a sub-agent — is **INVALID**: not judged, never a pass, exit 1. ### How a verdict is reached After each attempt a judge model (`co/gemini-3.8-flash` unless `--judge-model`) sees the case, the Agent's final answer and what its tools actually did — name, arguments, status, result — and answers one narrow question per statement: did this outcome **occur**, **not occur**, or **cannot be verified** from what the run shows? The mapping is code, not the judge: | statement | occurred | did not occur | cannot verify | |---|---|---|---| | `must` | PASS | FAIL | UNVERIFIED | | `must_not` | **FAIL** | PASS | UNVERIFIED | An outcome that changes the outside world — sent, submitted, saved, paid — has occurred only if a tool result shows it. An Agent that only *says* it submitted something gets UNVERIFIED, never PASS. ### Side effects A benchmark run must not become a production action. Without `--live`, the run sets `CO_EVAL_LIVE=0` in the environment for its whole duration; tools and skills that can rehearse should check it, and the judge still refuses to count an unproven effect as done. `--live` sets `CO_EVAL_LIVE=1`. The flag does not itself stop a tool that ignores it: point the Agent at fixtures or a safe environment, as the demo below does with a local ledger. ### Exit codes | exit | meaning | |---|---| | 0 | every expectation passed and, with `--skill`, the skill ran every time | | 1 | any FAIL, UNVERIFIED, STOPPED, INVALID, or a skill that did not run | | 2 | bad benchmark, agent path, skill or option — nothing was run | | 3 | the Agent or the runner broke — never counted as a pass | ## Reports Every run is a new directory, never overwritten: ```text .co/eval-runs/// ├── report.json read-only: agent, model, skill path + hash, invoke mode, every verdict with its reason └── traces/-.json the full trace of each attempt ``` The authored benchmark is never written to. `co eval report ` reopens the latest run (or `--run ID`) and compares it with the one before: score, cases newly passing or failing, newly forbidden outcomes, and a warning when the benchmark file, the invoke mode, the model or the agent changed — so a changed setup is not read as a changed skill. The score is passed checks over checks, where the checks are every expectation plus "did the skill run". ## What it looked like, 24 September 2026 A reimbursement Agent on `co/gemini-3.8-flash`, five cases, the submit tool writing to a local ledger: | run | skill | invoke | result | |---|---|---|---| | 1 | naive ("submit everything, say it is paid") | auto | 0/5 cases — every expectation passed, but the skill never ran once | | 2 | naive | explicit | 3/5 — INV-302 submitted, the duplicate submitted twice, "paid" claimed: 3 forbidden | | 3 | rules for title, amount and duplicates | explicit | **5/5**, 50% → 100%, "newly passing: duplicate-invoice, mixed-batch-title-mismatch" | Only `SKILL.md` changed between runs 2 and 3; every verdict matched the ledger. Run 1 is the one worth remembering: the model already did the right thing, and the benchmark still failed it, because what was being measured was the skill. ## The older `co eval` `co eval` and `co eval ` still run `.co/evals/*.yaml` exactly as before ([docs/debug/eval.md](../debug/eval.md)). A first word that is not `run` or `report` goes there, so no existing command changes meaning, and those files are never read as benchmarks.