Evaluation

eval/gold.yaml holds 21 adversarially verified questions (layers nightmares, multi-face cards, errata traps, Commander, out-of-scope) with expected rule ids and reference answers.

cargo run -p judge-eval -- recall            # retrieval gate: expected rules present in context? (free)
cargo run -p judge-eval -- answer --label x --limit 21 --max-usd 6   # full live run (~$2.50)
cargo run -p judge-eval -- rescore eval/runs/x.json                  # re-grade a stored run (free)
cargo run -p judge-eval -- show eval/runs/x.json                     # bot vs. gold, side by side

Scoring accepts per-question equivalence lists of alternate rule ids that state the same fact, so the metric tracks correctness rather than one author’s citation taste.

Results

Two full runs of the 21 questions, 2026-09-20, CR 2026-08-19, Voyage voyage-3.5 embeddings. The run files are committed under eval/published/ with every question, answer, citation, time and cost, so none of this has to be taken on trust: judge-eval show eval/published/v1-opus-5.json prints each answer beside its reference.

	claude-opus-5, both stages (the default)	claude-sonnet-5, both stages
Out-of-scope questions declined (of 3)	3	3
In-scope questions answered (of 18)	17	6
…agreeing with the reference ruling	17	5
…partly (right on the main point, a sub-question missed)	0	1
…contradicting the reference	0	0
Asked “did you mean?” instead	1	1
Not answered	0	11
Expected rule ids cited	45 of 67 (67%)	12 of 67 (18%)
Cost per in-scope question (median)	$0.13	$0.10
Cost per question answered	$0.14	$0.30
Time per in-scope question (median / longest)	22 s / 43 s	26 s / 589 s
Whole run	$2.39	$1.83

What these measure, and what they do not:

• Answered means a verdict that passed validation: every citation names a source the model was shown and quotes it verbatim, and every rule number in the text is one of those citations. “Not answered” is the pipeline refusing to show an answer, not a wrong answer shown. Sonnet’s eleven were five answers with no citations or no text, four errors in the tool round (a second lookup_rules request, a malformed rule id, a request the API refused), a bad quote, and one answer naming rules it did not cite.
• “Did you mean?” is the pipeline working as designed and an eval question unanswered. On Opus the extractor passed [[bob]] through in brackets, and a bracketed name only ever matches exactly. On Sonnet it offered “Bruna” and “Gisela” as written, each of which is several cards.
• Agreement with the reference was judged by Claude reading each answer against the gold set’s reference answer under a strict rubric. The references were written and checked by models, then audited against Oracle text, rulings and CR text (which found three to correct). No human judge has reviewed either side, so read this column as “no contradiction found”, not as a measured accuracy.
• Expected rule ids cited tracks how closely the citations match the gold set’s lists, which include background rules a good answer may leave out. It is a floor on citation overlap and a regression signal between runs, not an accuracy score.
• Twenty-one questions chosen to be hard is a small, adversarial sample. It says the pipeline holds up on layers, multi-faced cards, old wordings and Commander. It does not say how often an answer in your server will be right.
• The Sonnet dollars err high. That run’s configuration (eval/published/v1-sonnet-5.judge.toml) prices input and output at list and leaves cache reads at the input price, which is how an unlisted price defaults. The counts of questions answered do not depend on it.

The prompts are tuned on the default, and the second column is what that costs a smaller model: Sonnet 5 runs at 40% of the token price, fails validation or the tool round on most hard questions and pays for the retries anyway. There is no cheaper configuration to recommend on this evidence. The documentation site’s Model choice page covers what does save money.