Explainer 3: Failure, feedback and money

Part three of the explainer.

7. Failure catalogue

The tables below list the failure classes this kind of system has. Each row says where the handling lives, so you can read further.

Model behaviour

Problem	Handling
Model invents rule numbers or misquotes	citation validation, and the source’s own span is stored (core/verdict.rs, core/quote.rs)
Model answers from memory instead of the material	system prompt ground rule 1, required citations, retry notice
Model pads with placeholder citations	prompt forbids stubs; an entry that quotes nothing (blank, "placeholder", a character or two) is dropped and the rest of the answer validated as usual, and an answer with nothing but stubs is rejected with the parse error shown back. A malformed entry that does quote something is still a typed rejection
Answer names a rule number it never cited (“per 605.3b…”)	every rule number in the prose must be covered by a rule citation (the id, its rule, or a sub-rule); otherwise the attempt is rejected and the retry is told to cite it or remove it
Model files a card’s Oracle text as a ruling (the card has no rulings to cite)	still rejected, never relabelled; the retry notice names the kind it meant (oracle_text, with the card and face) instead of telling it to drop a good quote
Model calls the tool repeatedly	Synth typestate: one round, by type
Model’s output is cut off at max_tokens	detected from the stop reason, retried once at medium effort
Model claims a question is out of scope to dodge citing	source is stamped from extraction, not model-reported
Model is asked about tournament policy	classifier routes Tournament/OutOfScope to a decline before any synthesis spend

Card names

Problem	Handling
Nicknames (“bob”, “goyf”)	curated alias table, possessive stripping
Old or errata’d names	printed_names from every printing
Typos	trigram fuzzy with a margin rule
Two cards could be meant	Resolution::Ambiguous → “did you mean?” buttons, never guessed
User wrote both nickname and full name	duplicate-span detection in core/judge.rs
Rules vocabulary mistaken for a card (“trample”)	extraction runs first, so fuzzy sees only card spans
Card text has changed since an answer was stored	Oracle fingerprints on each call, and the retirement pass (db/retire.rs)

Retrieval

Problem	Handling
Right rule uses different words than the question	vector leg
Vector leg returns thematically near but wrong rules	union with the exact legs, and the model can lookup_rules
Classifier picks the wrong category	full-text and vector legs, lookup_rules
Too much material for the prompt	Budget in bot/synth.rs, where the structured leg survives cuts
Model cites a sub-rule shown only inside its parent	the synthesizer hydrates the leaf row so validation finds it
CR renumbered, so stored calls cite stale ids	renumber_map (ingest, §8)
A cited rule was reworded or deleted	retirement pass marks the call retired, and restores it if the text returns

Vectors

Problem	Handling
Vectors from two embedding models in one column (silently wrong results)	embedding_space table names the model, and readers and writers check it on every use (db/space.rs)
Embedding model switched while the bot runs	vector legs go dark with an error log rather than mixing spaces
Switching models is expensive (re-pays every row)	ingest reembed --yes is explicit, probes the new embedder first, and prints a rough cost without --yes
A switch races an in-flight write	advisory lock: writers take the shared side, the switch takes the exclusive side
No embedding key configured	vector leg off, and the other legs still work
Voyage free-tier token limits	batch size knob VOYAGE_MAX_BATCH
HNSW post-filtering shrinking results	partial index over the rows that are searched and no others

Money and abuse

Problem	Handling
Runaway API spend	Metered spend cap by reservation (§9), the only ChatModel there is
Many concurrent requests	a semaphore (JUDGE_CONCURRENCY) shared by web and MCP
Anonymous web abuse	per-IP fixed-window rate limit, bucketed on an address the caller cannot forge
Leaked MCP token	separate per-window limit on judge runs through /mcp
Agent sessions reading Discord history	AgentThread ids are a type that can only be agent:<uuid>
Oversized agent inputs	bounded question, span, id and answer lengths

Operations

Problem	Handling
New CR release	nightly scrape of Wizards’ page, version compared before download
Scryfall data drift	nightly bulk re-sync, with content-hashed ruling keys that keep identity
Prompt or schema drift breaking the wire format	golden request fixtures pinned byte-for-byte, and the system prompt SHA pinned
Losing the database (re-embedding costs money)	weekly pg_dump to R2 with a restore drill
Unknown config keys silently ignored	deny_unknown_fields and “this knob would be ignored” errors at load

---

8. Feedback loop and stale answers

Stored answers are an asset (examples for future questions) and a liability (they go stale). Two mechanisms keep them current without a human curator.

Retirement. Every call’s citations are its declared dependencies on the world. The nightly pass re-runs the same substring check that admitted each citation, against today’s rules, rulings and Oracle text. If any check fails, the call is retired and leaves the prior-call leg. If the text comes back, the call is restored. Each call also stores a fingerprint of the Oracle text of every card in its context, so an erratum retires calls about a card even when they cited only the CR. This replaced an earlier rule that retired every call on every CR release. That rule threw away many still-correct answers and kept wrong ones after an erratum.

Renumbering. When Wizards inserts a keyword at 702.20, every later rule shifts by one. Their bodies change too, because cross-references shift with them. Comparing the raw text fails on the very release it is meant to see through. The CR loader masks every rule id out of every body and matches old and new rules on the masked text where it is unique on both sides. It then rewrites each old rule with the full map and keeps only mappings that reproduce the new rule exactly. This consistency check rejects a cross-reference that was redirected rather than renumbered. Matched calls get their citation ids, quoted ids and answer text rewritten in one pass. Anything ambiguous is left for the retirement pass to judge. The principle is the same as card resolution: never guess.

---

9. Spend cap

Every model call in every binary goes through one Metered wrapper around the backend. The wrappers in a process share one SpendMeter with a cap (JUDGE_MAX_USD, default $5). Before a request is sent, its worst-case cost is reserved against the counter: the request at the input price plus max_tokens at the output price. If that would breach the cap, the request is refused. After the response, the reservation is replaced with the actual usage. So concurrent callers cannot collectively overshoot, and a response body that fails to decode is still billed when its usage could be read.

The trait the pipeline calls (ChatModel) is sealed and Metered is its only implementor, so a backend that skips the cap cannot be handed to the pipeline. A local model priced Free is counted but never refused.

Two prompt-caching details cut cost. The system prompts are stable and carry a cache breakpoint, so repeated questions reuse the cached prefix. The rendered material in the synthesis user turn has its own breakpoint, so the tool-round continuation rereads it at the cache price. After a tool round the tool_choice stays auto rather than switching to none, because changing it would invalidate that cache.

A full run of the 21-question gold evaluation set costs about $2.50. Development is done against a mocked HTTP server (wiremock), not the live API.

---

10. The three front doors

All three share one composition root, judge_bot::build_deps, so they run the same pipeline.

• Discord (crates/bot): a /judge slash command, thread history as context, “did you mean?” buttons backed by a pending store, rating buttons, card mana symbols drawn as application emoji. The rendering logic is pure and unit-tested. A mana emoji tag is about thirty characters and must never be cut in half by Discord’s length limit, so rendered text is carried as segments where only plain text is cuttable.
• Web (crates/api + web/, a SolidJS page): anonymous, so no ratings. Ambiguity comes back as data. The client re-asks with pins that the server rewrites to [[Full Card Name]]. Session history keys on a client UUID. Rate-limited per IP.
• Agent (crates/agent, judge-cli and judge-mcp): the judge as a tool for other AI agents. It has two modes. The judge tool runs the pipeline as above with the built-in model. A session runs it in pull mode, where the outside agent is the model. It receives the extraction prompt, returns extraction JSON, receives the rendered synthesis prompt, and returns a verdict. That verdict goes through the same validation. State lives in Postgres between calls as a Stage enum, with the same one-tool-round, one-retry limits. This is also the cheapest way to reproduce a bad answer: Claude Code drives it directly (.claude/skills/judge/SKILL.md), spending nothing.

---