Skip to content

Explainer 1: What it does and why

The first of four parts of docs/EXPLAINER.md, an end-to-end explanation for a programmer new to retrieval-augmented systems.

A user in a Discord server (or in a web app, or an AI agent over MCP) asks a Magic: The Gathering rules question:

Does bob’s trigger still happen if he dies in response to it?

The bot answers with the ruling first, then the reasoning, with rule numbers. Every answer carries citations. Each citation is a short verbatim quote from one of four kinds of source:

  • a numbered rule in the Comprehensive Rules (the CR, a large text file Wizards publishes),
  • a Scryfall ruling (official card-specific clarifications),
  • a card’s current Oracle text (its official wording, which changes through errata),
  • a prior call the bot itself made earlier that users rated well.

Discord users rate each answer 1 to 3, and so, anonymously, can web visitors. Ratings never make the bot “learn” in the model sense. Discord’s decide which old answers the model sees later as examples, and nothing else. The web’s are only counted and shown. A rating that finds fault comes with a reason (what kind of problem, what is wrong, the right answer), which goes to the operator’s feedback queue, not to the model.

The system is one Postgres database, a handful of Rust binaries, two paid APIs (a chat model for reasoning, an embedding model for search), and a daily data refresh the long-running processes run themselves.


A large language model already “knows” a lot about Magic. Asking it directly fails in three ways that matter for a judge bot:

  1. It is confidently wrong. Rule numbers get invented, old Oracle text gets quoted as current, and a plausible paragraph looks the same as a correct one.
  2. Its knowledge is frozen. New sets, keywords and errata arrive monthly. The CR is renumbered a few times a year.
  3. Nobody can check it. An answer without a pointer into the rules text cannot be verified by the asker or by a human judge.

The standard remedy is retrieval-augmented generation (RAG). Before asking the model, look up the relevant source material yourself and hand it to the model. Tell the model to answer only from that material. This bot is a RAG system with two additions that most RAG systems skip:

  • Citation validation. The model must quote its sources, and the program checks that every quote is a substring of the source it names. A failed check rejects the answer.
  • Card lookup before search (entity resolution). Card names are looked up in a table, not searched for by meaning. “bob” must become one card, Dark Confidant, or the user must be asked.

Take the question above. The steps below happen in order. The pipeline lives in crates/core/src/judge.rs. Each step is a trait (a “port”) defined in core and implemented by an adapter in crates/bot.

This step is one cheap model call. The bot sends the question, plus the last five Q&A pairs in the same thread, to the chat model with a small system prompt (crates/bot/src/prompts/extract_system.md, the taxonomy filled in from data/categories.yaml) and a JSON schema for the reply. The model returns:

{
"card_spans": ["bob"],
"concepts": ["dies in response to trigger", "leaves-the-battlefield trigger"],
"primary": {"category": "triggered_abilities", "confidence": "high"},
"secondary": [{"category": "zones", "confidence": "low"}],
"source": "cr"
}

Three things happen in this call:

  • Card spans, the words that name cards, are cut out of the sentence. Fuzzy name matching later sees bob alone. It never sees “trigger” or “response”, which would otherwise fuzzy-match real cards.
  • The question is classified into a fixed taxonomy of 25 categories that mirror the CR’s structure (data/categories.yaml). primary is required. Up to two secondary guesses are kept. The categories drive the first retrieval source.
  • The source says whether this is a rules question (cr), a Commander-format question, tournament policy, or off-topic. The last two stop here with a polite decline.

“Structured output” means the API constrains the model to emit JSON matching a schema. The schema is generated from the Rust struct the reply is decoded into, so the two cannot drift apart.

This step uses SQL and no model. Each span goes down a resolution order of increasingly loose lookups and stops at the first step that answers:

  1. hand-curated alias table (bob → Dark Confidant),
  2. the same after stripping a possessive (bob's), also retrying the exact and short-name steps,
  3. exact current name (also matches a single face of a two-faced card),
  4. every name the card has ever been printed under (old names, errata’d names),
  5. the part of a name before the comma (Jace → several Jaces → ambiguous),
  6. the part after it, when it is two words or more (Nimble Pilferer → Ragavan), unless another card’s name holds the same words (Grand Arbiter → Dovin, Grand Arbiter or Grand Arbiter Augustin IV? → ambiguous),
  7. a nickname preceded only by printing words (foil bob),
  8. trigram fuzzy match, for typos.

A span written in brackets, [[Full Card Name]], skips that resolution order. The brackets say “this exact name”, so it is tried only against current and printed names, then the alias table. An alias names one card for that exact spelling, so [[bob]] is Dark Confidant, as bob is. A looser match is offered as a question, never taken as the answer. [[bob's]] asks “did you mean Dark Confidant?”. It asks nothing when the extractor already named Dark Confidant from the same question. A near miss like [[Dark Confidnt]] offers the closest spellings. Answers name the cards they resolved to (“Cards: …”), so the reader can see what a nickname was taken to mean.

The resolver never guesses. If two or more cards remain, the result is Ambiguous and Discord shows “Did you mean…?” buttons. If nothing matches, the result is NotFound. The type system forces every consumer to handle all three outcomes.

Fuzzy matching uses Postgres’s pg_trgm extension. A trigram is a three-letter window. “bolt” is {" b","bo","ol","lt","t "}. Two strings are similar when they share many trigrams, which tolerates typos without any model. The fuzzy step accepts a candidate in two cases: it is alone with a strong score (0.7 or more), or it leads the runner-up by a clear margin (0.15).

Now the bot assembles the “material”: everything the model will be allowed to read. It runs seven queries concurrently and unions the results into a Context:

  • CR rules, from three retrieval sources (explained in §5): the curated subsections for the categories, a full-text keyword search, and a vector similarity search.
  • Scryfall rulings for every face of every resolved card.
  • Glossary entries whose term appears in the cards’ Oracle text.
  • Nightmare-card notes: hand-written explanations for cards like Humility whose interactions are counterintuitive.
  • Prior calls: up to five earlier well-rated answers in the same categories about one of the same cards (any card, if none resolved).

The thread history is added to the context as well, so “what if it also had flying?” makes sense.

This step is one expensive model call with one optional tool round. The context is rendered into a long user turn under a character budget:

  • 25 rule chunks,
  • 30,000 characters of rules text,
  • 20 rulings per card,
  • 4 history pairs, with each earlier answer cut to 600 characters.

It is sent with the judge system prompt in crates/bot/src/prompts/synth_system.md. That file is worth reading, because it is the contract between program and model.

The model may make at most one lookup_rules tool call to fetch rules the retrieval missed (“I need 603.10”). Then it must answer with a JSON verdict:

{
"answer": "Yes. Dark Confidant's trigger ...",
"confidence": "high",
"category": "triggered_abilities",
"citations": [
{"kind": "rule", "id": "603.10", "quote": "Normally, objects that exist immediately after an event are checked..."},
{"kind": "oracle_text", "card": "9f2c...", "face": 0, "quote": "At the beginning of your upkeep, reveal the top card"}
]
}

The other two citation kinds are scryfall_ruling (card, ruling, quote) and prior_call (id, quote).

For every citation the program checks two things. The id must name something that was in the context, and the quote must be a substring of that source’s text. If anything fails, or the verdict has no citations, the model gets one retry with the rejection rendered into the prompt (“your quote ... was not found in 603.10”). A second failure is an error reply.

The validated verdict is a different Rust type from the raw one. Only the validated type can be saved or sent to Discord (§6).

The call is stored with its question, answer, citations, the ids of everything in its context, and the CR version it was answered under. Discord gets the answer with rule numbers linked to a CR mirror, card symbols drawn as emoji, and 1/2/3 rating buttons.