Skip to content

Explainer 3: Failure, feedback and money

Part three of the explainer.

The tables below list the failure classes this kind of system has. Each row says where the handling lives, so you can read further.

Problem Handling
Model invents rule numbers or misquotes citation validation, and the source’s own span is stored (core/verdict.rs, core/quote.rs)
Model answers from memory instead of the material system prompt ground rule 1, required citations, retry notice
Model pads with placeholder citations the prompt forbids stubs. An entry that quotes nothing (blank, "placeholder", a character or two) is dropped, and the rest of the answer is validated as usual. An answer with nothing but stubs is rejected, with the parse error shown back. A malformed entry that does quote something is still a typed rejection
Answer names a rule number it never cited (“per 605.3b…”) every rule number in the prose must be covered by a rule citation: the id, its rule, or a sub-rule. Otherwise the attempt is rejected, and the retry is told to cite it or remove it
Model files a card’s Oracle text as a ruling (the card has no rulings to cite) still rejected, never relabelled. The retry notice names the kind it meant (oracle_text, with the card and face) instead of telling it to drop a good quote
Model calls the tool repeatedly Synth typestate: one round, by type
Model’s output is cut off at max_tokens detected from the stop reason, retried once at medium effort, or low from medium
Model claims a question is out of scope to dodge citing source is stamped from extraction, not model-reported
Model is asked about tournament policy classifier routes Tournament/OutOfScope to a decline before any synthesis spend
Problem Handling
Nicknames (“bob”, “goyf”) curated alias table, possessive stripping
Old or errata’d names printed_names from every printing
Typos trigram fuzzy with a margin rule
Two cards could be meant Resolution::Ambiguous → “did you mean?” buttons, never guessed
User wrote both nickname and full name duplicate-span detection in core/judge.rs
Rules vocabulary mistaken for a card (“trample”) extraction runs first, so fuzzy sees only card spans
Card text has changed since an answer was stored Oracle fingerprints on each call, and the retirement pass (db/retire.rs)
Problem Handling
Right rule uses different words than the question vector search
Vector search returns thematically near but wrong rules union with the exact sources, and the model can lookup_rules
Classifier picks the wrong category full-text and vector search, lookup_rules
Too much material for the prompt Budget in bot/synth.rs, where the structured source survives cuts
Model cites a sub-rule shown only inside its parent the synthesizer hydrates the leaf row so validation finds it
CR renumbered, so stored calls cite stale ids renumber_map (ingest, §8)
A cited rule was reworded or deleted retirement pass marks the call retired, and restores it if the text returns
Problem Handling
Vectors from two embedding models in one column (silently wrong results) embedding_space table names the model, and readers and writers check it on every use (db/space.rs)
Embedding model switched while the bot runs vector search is turned off with an error log rather than mixing spaces
Switching models is expensive (re-pays every row) ingest reembed --yes is explicit, probes the new embedder first, and prints a rough cost without --yes
A switch races an in-flight write advisory lock: writers take the shared side, the switch takes the exclusive side
No embedding key configured vector search off, and the other sources still work
Voyage free-tier token limits batch size setting VOYAGE_MAX_BATCH
HNSW post-filtering shrinking results partial index over the rows that are searched and no others
Problem Handling
Runaway API spend Metered spend cap by reservation (§9), the only ChatModel there is
Many concurrent requests a semaphore (JUDGE_CONCURRENCY) shared by web and MCP
Anonymous web abuse per-IP fixed-window rate limit, bucketed on an address the caller cannot forge
Leaked MCP token separate per-window limit on judge runs through /mcp
Agent sessions reading Discord history AgentThread ids are a type that can only be agent:<uuid>
Oversized agent inputs bounded question, span, id and answer lengths
Problem Handling
New CR release daily scrape of Wizards’ page, version compared before download
Scryfall data drift daily bulk re-sync, with content-hashed ruling keys that keep identity
Prompt or schema drift breaking the wire format golden request fixtures pinned byte-for-byte, and the system prompt SHA pinned
Losing the database (re-embedding costs money) weekly pg_dump to an S3-compatible bucket with a restore drill
Unknown config keys silently ignored deny_unknown_fields and “this option would be ignored” errors at load

Stored answers are an asset (examples for future questions) and a liability (they go stale). Two mechanisms keep them current without a human curator, and a third hands the rest to one.

Retirement. Every call’s citations name the text it depends on. The refresh’s retirement pass re-runs the same substring check that admitted each citation, against today’s rules, rulings and Oracle text. If any check fails, the call is retired and leaves the prior-call query. If the text comes back, the call is restored. Each call also stores a fingerprint of the Oracle text of every card in its context, so an erratum retires calls about a card even when they cited only the CR. This replaced an earlier rule that retired every call on every CR release. That rule threw away many still-correct answers and kept wrong ones after an erratum.

Renumbering. When Wizards inserts a keyword at 702.20, every later rule shifts by one. Their bodies change too, because the cross-references inside them shift as well, so comparing the raw text fails on the release it is meant to handle. Instead, the CR loader:

  1. masks every rule id out of every rule’s text,
  2. matches old and new rules on the masked text, only where that text is unique on both sides,
  3. rewrites each old rule with the full old-to-new map, and keeps only matches that then reproduce the new rule exactly.

Step 3 rejects a cross-reference that was pointed at a different rule rather than renumbered. Matched calls get their citation ids, quoted ids and answer text rewritten in one pass. Anything ambiguous is left for the retirement pass to judge. As in card resolution, the loader does not guess.

The feedback queue. Some wrong answers are not stale, just wrong: the card was misread, the right rule never retrieved. A Partially correct or Incorrect rating asks the rater why, and each reason becomes an item in a queue the operator works from the admin page, judge-cli or an agent’s MCP tools, beside the call’s full material. The fix lands in the data (a card note, a nickname) or the code (retrieval, a prompt), and the item is resolved with a note saying what was done. A rater who replaces their reason reopens it.


Every model call in every binary goes through one Metered wrapper around the backend. The wrappers in a process share one SpendMeter with a cap (JUDGE_MAX_USD, default $5). Before a request is sent, its worst-case cost is reserved against the counter: the request at the input price plus max_tokens at the output price. If that would breach the cap, the request is refused. After the response, the reservation is replaced with the actual usage. So concurrent callers cannot collectively overshoot, and a response body that fails to decode is still billed when its usage could be read.

The trait the pipeline calls (ChatModel) is sealed and Metered is its only implementor, so a backend that skips the cap cannot be handed to the pipeline. A local model priced Free is counted but never refused.

Embeddings bill to the same meter. The embedder the retriever, the call store and the ingest steps take (WithSpace) is sealed too, implemented only by MeteredEmbedder. It reserves a text’s bytes plus a small allowance as tokens, since no tokenizer makes a token of less than a byte, and settles to the provider’s reported usage. A refused embedding costs a question its vector search, never its answer.

Two prompt-caching details cut cost. The system prompts are stable and carry a cache breakpoint, so repeated questions reuse the cached prefix. The rendered material in the synthesis user turn has its own breakpoint, so the tool-round continuation rereads it at the cache price. After a tool round the tool_choice stays auto rather than switching to none, because changing it would invalidate that cache.

A full run of the 24-question gold evaluation set costs about $1.85. Development is done against a mocked HTTP server (wiremock), not the live API.


All three share one composition root, judge_bot::build_deps, so they run the same pipeline.

  • Discord (crates/bot): a /judge slash command, thread history as context, “did you mean?” buttons answered from the message they are on, rating buttons, card mana symbols drawn as application emoji. The rendering logic is pure and unit-tested. A mana emoji tag is about thirty characters and must never be cut in half by Discord’s length limit, so rendered text is carried as segments where only plain text is cuttable.
  • Web (crates/api + web/, a SolidJS web app): anonymous, so its ratings never reach retrieval; a reason given with one goes to the operator’s queue. It also serves the public feed of answered questions and a dashboard of their counts. Ambiguity comes back as data. The client re-asks with pins that the server rewrites to [[Full Card Name]]. Session history keys on a client UUID. Rate-limited per IP.
  • Agent (crates/agent, judge-cli and judge-mcp): the judge as a tool for other AI agents. It has two modes. The judge tool runs the pipeline as above with the built-in model. A session runs it in pull mode, where the outside agent is the model. It receives the extraction prompt, returns extraction JSON, receives the rendered synthesis prompt, and returns a verdict. That verdict goes through the same validation. State lives in Postgres between calls as a Stage enum, with the same one-tool-round, one-retry limits. This is also the cheapest way to reproduce a bad answer: Claude Code drives it directly (.claude/skills/judge/SKILL.md), spending nothing.