Explainer 3: Failure, feedback and money
Part three of the explainer.
7. Failure catalogue
Section titled “7. Failure catalogue”The tables below list the failure classes this kind of system has. Each row says where the handling lives, so you can read further.
Model behaviour
Section titled “Model behaviour”| Problem | Handling |
|---|---|
| Model invents rule numbers or misquotes | citation validation, and the source’s own span is stored (core/verdict.rs, core/quote.rs) |
| Model answers from memory instead of the material | system prompt ground rule 1, required citations, retry notice |
| Model pads with placeholder citations | the prompt forbids stubs. An entry that quotes nothing (blank, "placeholder", a character or two) is dropped, and the rest of the answer is validated as usual. An answer with nothing but stubs is rejected, with the parse error shown back. A malformed entry that does quote something is still a typed rejection |
Answer names a rule number it never cited (“per 605.3b…”) |
every rule number in the prose must be covered by a rule citation: the id, its rule, or a sub-rule. Otherwise the attempt is rejected, and the retry is told to cite it or remove it |
| Model files a card’s Oracle text as a ruling (the card has no rulings to cite) | still rejected, never relabelled. The retry notice names the kind it meant (oracle_text, with the card and face) instead of telling it to drop a good quote |
| Model calls the tool repeatedly | Synth typestate: one round, by type |
Model’s output is cut off at max_tokens |
detected from the stop reason, retried once at medium effort, or low from medium |
| Model claims a question is out of scope to dodge citing | source is stamped from extraction, not model-reported |
| Model is asked about tournament policy | classifier routes Tournament/OutOfScope to a decline before any synthesis spend |
Card names
Section titled “Card names”| Problem | Handling |
|---|---|
| Nicknames (“bob”, “goyf”) | curated alias table, possessive stripping |
| Old or errata’d names | printed_names from every printing |
| Typos | trigram fuzzy with a margin rule |
| Two cards could be meant | Resolution::Ambiguous → “did you mean?” buttons, never guessed |
| User wrote both nickname and full name | duplicate-span detection in core/judge.rs |
| Rules vocabulary mistaken for a card (“trample”) | extraction runs first, so fuzzy sees only card spans |
| Card text has changed since an answer was stored | Oracle fingerprints on each call, and the retirement pass (db/retire.rs) |
Retrieval
Section titled “Retrieval”| Problem | Handling |
|---|---|
| Right rule uses different words than the question | vector search |
| Vector search returns thematically near but wrong rules | union with the exact sources, and the model can lookup_rules |
| Classifier picks the wrong category | full-text and vector search, lookup_rules |
| Too much material for the prompt | Budget in bot/synth.rs, where the structured source survives cuts |
| Model cites a sub-rule shown only inside its parent | the synthesizer hydrates the leaf row so validation finds it |
| CR renumbered, so stored calls cite stale ids | renumber_map (ingest, §8) |
| A cited rule was reworded or deleted | retirement pass marks the call retired, and restores it if the text returns |
Vectors
Section titled “Vectors”| Problem | Handling |
|---|---|
| Vectors from two embedding models in one column (silently wrong results) | embedding_space table names the model, and readers and writers check it on every use (db/space.rs) |
| Embedding model switched while the bot runs | vector search is turned off with an error log rather than mixing spaces |
| Switching models is expensive (re-pays every row) | ingest reembed --yes is explicit, probes the new embedder first, and prints a rough cost without --yes |
| A switch races an in-flight write | advisory lock: writers take the shared side, the switch takes the exclusive side |
| No embedding key configured | vector search off, and the other sources still work |
| Voyage free-tier token limits | batch size setting VOYAGE_MAX_BATCH |
| HNSW post-filtering shrinking results | partial index over the rows that are searched and no others |
Money and abuse
Section titled “Money and abuse”| Problem | Handling |
|---|---|
| Runaway API spend | Metered spend cap by reservation (§9), the only ChatModel there is |
| Many concurrent requests | a semaphore (JUDGE_CONCURRENCY) shared by web and MCP |
| Anonymous web abuse | per-IP fixed-window rate limit, bucketed on an address the caller cannot forge |
| Leaked MCP token | separate per-window limit on judge runs through /mcp |
| Agent sessions reading Discord history | AgentThread ids are a type that can only be agent:<uuid> |
| Oversized agent inputs | bounded question, span, id and answer lengths |
Operations
Section titled “Operations”| Problem | Handling |
|---|---|
| New CR release | daily scrape of Wizards’ page, version compared before download |
| Scryfall data drift | daily bulk re-sync, with content-hashed ruling keys that keep identity |
| Prompt or schema drift breaking the wire format | golden request fixtures pinned byte-for-byte, and the system prompt SHA pinned |
| Losing the database (re-embedding costs money) | weekly pg_dump to an S3-compatible bucket with a restore drill |
| Unknown config keys silently ignored | deny_unknown_fields and “this option would be ignored” errors at load |
8. Feedback and stale answers
Section titled “8. Feedback and stale answers”Stored answers are an asset (examples for future questions) and a liability (they go stale). Two mechanisms keep them current without a human curator, and a third hands the rest to one.
Retirement. Every call’s citations name the text it depends on. The refresh’s retirement pass re-runs the same substring check that admitted each citation, against today’s rules, rulings and Oracle text. If any check fails, the call is retired and leaves the prior-call query. If the text comes back, the call is restored. Each call also stores a fingerprint of the Oracle text of every card in its context, so an erratum retires calls about a card even when they cited only the CR. This replaced an earlier rule that retired every call on every CR release. That rule threw away many still-correct answers and kept wrong ones after an erratum.
Renumbering. When Wizards inserts a keyword at 702.20, every later rule shifts by one.
Their bodies change too, because the cross-references inside them shift as well, so
comparing the raw text fails on the release it is meant to handle. Instead, the CR
loader:
- masks every rule id out of every rule’s text,
- matches old and new rules on the masked text, only where that text is unique on both sides,
- rewrites each old rule with the full old-to-new map, and keeps only matches that then reproduce the new rule exactly.
Step 3 rejects a cross-reference that was pointed at a different rule rather than renumbered. Matched calls get their citation ids, quoted ids and answer text rewritten in one pass. Anything ambiguous is left for the retirement pass to judge. As in card resolution, the loader does not guess.
The feedback queue. Some wrong answers are not stale, just wrong: the card was
misread, the right rule never retrieved. A Partially correct or Incorrect rating
asks the rater why, and each reason becomes an item in a queue the operator works from
the admin page, judge-cli or an agent’s MCP tools, beside the call’s full material. The
fix lands in the data (a card note, a nickname) or the code (retrieval, a prompt), and
the item is resolved with a note saying what was done. A rater who replaces their reason
reopens it.
9. Spend cap
Section titled “9. Spend cap”Every model call in every binary goes through one Metered wrapper around the backend.
The wrappers in a process share one SpendMeter with a cap (JUDGE_MAX_USD, default $5).
Before a request is sent, its worst-case cost is reserved against the counter: the
request at the input price plus max_tokens at the output price. If that would breach the
cap, the request is refused. After the response, the reservation is replaced with the
actual usage. So concurrent callers cannot collectively overshoot, and a response body that
fails to decode is still billed when its usage could be read.
The trait the pipeline calls (ChatModel) is sealed and Metered is its only implementor,
so a backend that skips the cap cannot be handed to the pipeline. A local model priced
Free is counted but never refused.
Embeddings bill to the same meter. The embedder the retriever, the call store and the
ingest steps take (WithSpace) is sealed too, implemented only by MeteredEmbedder. It
reserves a text’s bytes plus a small allowance as tokens, since no tokenizer makes a
token of less than a byte, and settles to the provider’s reported usage. A refused
embedding costs a question its vector search, never its answer.
Two prompt-caching details cut cost. The system prompts are stable and carry a cache
breakpoint, so repeated questions reuse the cached prefix. The rendered material in the
synthesis user turn has its own breakpoint, so the tool-round continuation rereads it at
the cache price. After a tool round the tool_choice stays auto rather than switching to
none, because changing it would invalidate that cache.
A full run of the 24-question gold evaluation set costs about $1.85. Development is done
against a mocked HTTP server (wiremock), not the live API.
10. The three interfaces
Section titled “10. The three interfaces”All three share one composition root, judge_bot::build_deps, so they run the same
pipeline.
- Discord (
crates/bot): a/judgeslash command, thread history as context, “did you mean?” buttons answered from the message they are on, rating buttons, card mana symbols drawn as application emoji. The rendering logic is pure and unit-tested. A mana emoji tag is about thirty characters and must never be cut in half by Discord’s length limit, so rendered text is carried as segments where only plain text is cuttable. - Web (
crates/api+web/, a SolidJS web app): anonymous, so its ratings never reach retrieval; a reason given with one goes to the operator’s queue. It also serves the public feed of answered questions and a dashboard of their counts. Ambiguity comes back as data. The client re-asks with pins that the server rewrites to[[Full Card Name]]. Session history keys on a client UUID. Rate-limited per IP. - Agent (
crates/agent,judge-cliandjudge-mcp): the judge as a tool for other AI agents. It has two modes. Thejudgetool runs the pipeline as above with the built-in model. A session runs it in pull mode, where the outside agent is the model. It receives the extraction prompt, returns extraction JSON, receives the rendered synthesis prompt, and returns a verdict. That verdict goes through the same validation. State lives in Postgres between calls as aStageenum, with the same one-tool-round, one-retry limits. This is also the cheapest way to reproduce a bad answer: Claude Code drives it directly (.claude/skills/judge/SKILL.md), spending nothing.
MTG Judgebot is unofficial Fan Content permitted under the Fan Content Policy. Not approved or endorsed by Wizards of the Coast. Portions of the materials used are property of Wizards of the Coast. ©Wizards of the Coast LLC.
The Comprehensive Rules come from Wizards of the Coast. Card data, rulings and card symbols come from Scryfall, which is not affiliated with this project. Rule links go to the Yawgatog mirror. License and attribution.