Explainer 1: What it does and why

The first of four parts of docs/EXPLAINER.md, an end-to-end tour for a programmer new to retrieval-augmented systems.

1. What it does

A user in a Discord server (or on a small web page, or an AI agent over MCP) asks a Magic: The Gathering rules question:

Does bob’s trigger still happen if he dies in response to it?

The bot answers like a judge would: ruling first, then the reasoning, with rule numbers. Every answer carries citations. Each citation is a short verbatim quote from one of four kinds of source:

• a numbered rule in the Comprehensive Rules (the CR, a large text file Wizards publishes),
• a Scryfall ruling (official card-specific clarifications),
• a card’s current Oracle text (its official wording, which changes through errata),
• a prior call the bot itself made earlier that users rated well.

Discord users rate each answer 1 to 3. Ratings never make the bot “learn” in the model sense. They decide which old answers the model sees later as examples, and nothing else.

The system is one Postgres database, a handful of Rust binaries, two paid APIs (a chat model for reasoning, an embedding model for search), and a nightly refresh job.

---

2. Limits of asking the model directly

A large language model already “knows” a lot about Magic. Asking it directly fails in three ways that matter for a judge bot:

1. It is confidently wrong. Rule numbers get invented, old Oracle text gets quoted as current, and a plausible paragraph looks the same as a correct one.
2. Its knowledge is frozen. New sets, keywords and errata arrive monthly. The CR is renumbered a few times a year.
3. Nobody can check it. An answer without a pointer into the rules text cannot be verified by the asker or by a human judge.

The standard remedy is retrieval-augmented generation (RAG). Before asking the model, look up the relevant source material yourself and hand it to the model. Tell the model to answer only from that material. This bot is a RAG system with two additions that most RAG systems skip:

• Citation validation. The model must quote its sources, and the program checks that every quote is a substring of the source it names. A failed check rejects the answer. This turns “the model was told to cite” into “the answer is grounded”.
• Entity resolution before search. Card names are looked up in a table, not searched for semantically. “bob” must become one card, Dark Confidant, or the user must be asked.

Most of this document is about how those two ideas play out.

---

3. One question, end to end

Take the question above. The steps below happen in order. The pipeline lives in crates/core/src/judge.rs. Each step is a “port” (a trait) that core defines and an adapter in crates/bot implements.

Step 1: Extraction and classification

This step is one cheap model call. The bot sends the question, plus the last five Q&A pairs in the same thread, to the chat model with a small system prompt and a JSON schema for the reply. The model returns:

{
  "card_spans": ["bob"],
  "concepts": ["dies in response to trigger", "leaves-the-battlefield trigger"],
  "primary": {"category": "triggered_abilities", "confidence": "high"},
  "secondary": [{"category": "zones", "confidence": "low"}],
  "source": "cr"
}

Three things happen in this call:

• Card spans are cut out of the sentence. Fuzzy name matching later sees bob alone. It never sees “trigger” or “response”, which would otherwise fuzzy-match real cards.
• The question is classified into a fixed taxonomy of 25 categories that mirror the CR’s structure (data/categories.yaml). primary is required. Up to two secondary guesses are kept. The categories drive the first retrieval leg.
• The source says whether this is a rules question (cr), a Commander-format question, tournament policy, or off-topic. The last two stop here with a polite decline.

“Structured output” means the API constrains the model to emit JSON matching a schema. The schema is generated from the Rust struct the reply is decoded into, so the two cannot drift apart.

Step 2: Card resolution

This step uses SQL and no model. Each span goes down a ladder of increasingly loose lookups and stops at the first rung that answers:

1. hand-curated alias table (bob → Dark Confidant),
2. the same after stripping a possessive (bob's), also retrying the exact and short-name rungs,
3. exact current name (also matches a single face of a two-faced card),
4. every name the card has ever been printed under (old names, errata’d names),
5. the part of a name before the comma (Jace → several Jaces → ambiguous),
6. a nickname preceded only by printing words (foil bob),
7. trigram fuzzy match, for typos.

A span written in brackets, [[Full Card Name]], skips that ladder. The brackets say “this exact name”, so it is tried only against current and printed names. Anything else is offered, never resolved. [[bolt]] asks “did you mean Lightning Bolt?”, and asks nothing when the extractor already named Lightning Bolt from the same question. A near miss like [[Dark Confidnt]] offers the closest spellings. Answers name the cards they resolved to (“Cards: …”), so the reader can see what a nickname was taken to mean.

The important property: it never guesses. If two or more cards remain, the result is Ambiguous and Discord shows “Did you mean…?” buttons. If nothing matches, the result is NotFound. The type system forces every consumer to handle all three outcomes.

Fuzzy matching uses Postgres’s pg_trgm extension. A trigram is a three-letter window. “bolt” is {" b","bo","ol","lt","t "}. Two strings are similar when they share many trigrams, which tolerates typos without any model. The fuzzy rung accepts a candidate in two cases: it is alone with a strong score (0.7 or more), or it leads the runner-up by a clear margin (0.15).

Step 3: Retrieval

Now the bot assembles the “material”: everything the model will be allowed to read. It runs seven queries concurrently and unions the results into a Context:

• CR rules, from three “legs” (explained in §5): the curated subsections for the categories, a full-text keyword search, and a vector similarity search.
• Scryfall rulings for every face of every resolved card.
• Glossary entries whose term appears in the cards’ Oracle text.
• Nightmare-card notes: hand-written explanations for cards like Humility that break everyone’s intuition.
• Prior calls: up to five earlier well-rated answers in the same categories about one of the same cards (any card, if none resolved).

The thread history is added to the context as well, so “what if it also had flying?” makes sense.

Step 4: Synthesis

This step is one expensive model call with one optional tool round. The context is rendered into a long user turn under a character budget:

• 25 rule chunks,
• 30,000 characters of rules text,
• 20 rulings per card,
• 4 history pairs, with each earlier answer cut to 600 characters.

It is sent with the judge system prompt in crates/bot/src/prompts/synth_system.md. That file is worth reading, because it is the contract between program and model.

The model may make at most one lookup_rules tool call to fetch rules the retrieval missed (“I need 603.10”). Then it must answer with a JSON verdict:

{
  "answer": "Yes. Dark Confidant's trigger ...",
  "confidence": "high",
  "category": "triggered_abilities",
  "citations": [
    {"kind": "rule", "id": "603.10", "quote": "Normally, objects that exist immediately after an event are checked..."},
    {"kind": "oracle_text", "card": "9f2c...", "face": 0, "quote": "At the beginning of your upkeep, reveal the top card"}
  ]
}

The other two citation kinds are scryfall_ruling (card, ruling, quote) and prior_call (id, quote).

Step 5: Validation

For every citation the program checks two things. The id must name something that was in the context, and the quote must be a substring of that source’s text. If anything fails, or the verdict has no citations, the model gets one retry with the rejection rendered into the prompt (“your quote ... was not found in 603.10”). A second failure is an error reply.

The validated verdict is a different Rust type from the raw one. Only the validated type can be saved or sent to Discord (§6).

Step 6: Persist and reply

The call is stored with its question, answer, citations, the ids of everything in its context, and the CR version it was answered under. Discord gets the answer with rule numbers linked to a CR mirror, card symbols drawn as emoji, and 1/2/3 rating buttons.

---