Explainer 2: Data and retrieval
Part two of the explainer. Part one covered the pipeline end to end.
4. The data
Section titled “4. The data”Everything lives in one Postgres 16 database with two extensions: pgvector (vector
columns and indexes) and pg_trgm (trigram similarity). Migrations are in
crates/bot/migrations/.
| Table | Source | Refreshed | Notes |
|---|---|---|---|
cards, card_faces |
Scryfall bulk oracle_cards |
daily | one row per Oracle identity, and faces hold the Oracle text |
printed_names |
Scryfall bulk default_cards |
daily | every name ever printed, for old or errata’d names |
rulings |
Scryfall bulk rulings |
daily | keyed by a hash of the content, so a re-import is the same ruling |
rules, glossary |
the CR .txt from Wizards |
on release, detected daily | see chunking below |
card_aliases |
data/aliases.yaml |
when the built-in copy changes, detected daily | nicknames |
card_notes |
data/notes.yaml |
when the built-in copy changes, detected daily | nightmare cards |
curated_lists |
each alias or note load | with the list | where each list came from (built-in or a file), and its digest |
categories |
data/categories.yaml, via the compiled enum |
with each CR load | category → CR subsections |
calls, ratings |
every interface | continuous | answers and their votes |
embedding_space |
ingest embed |
on switch | one row: which embedding model the vectors came from |
agent_sessions |
the agent surface | continuous | state for the step-by-step agent mode |
Scale is small: ~30k cards, ~2k rule chunks, well under 10k calls, so nothing here needs a dedicated vector database.
How the CR is chunked
Section titled “How the CR is chunked”A text file is useless to search until it is cut into pieces (“chunks”). Where to cut matters more than any other choice in a RAG system, because a piece is what gets found, what gets shown, and what gets cited.
The CR is a numbered hierarchy: section 702 (Keyword Abilities) → rule 702.19
(Trample) → sub-rules 702.19a, 702.19b. The parser emits rows at two granularities:
- Rule-level rows (
702.19): the body is the rule’s own sentence plus every lettered sub-rule, eachExample:paragraph directly under the line it belongs to. These are the search unit. They get embeddings and appear in retrieval results. A rule plus its sub-rules is usually one coherent idea of a few hundred words, the right size for a model to read in one piece. - Leaf rows (
702.19b): one line each, withparent_id = 702.19. These are the citation unit. A model that quotes sub-rule b should cite702.19b, not the rule above it.
Scoring and lookups treat a leaf and its parent as covering each other. No rows exist for
three-digit sections. Asking for 702 expands to every rule in it.
5. Retrieval
Section titled “5. Retrieval”The bot combines three search techniques because each fails differently.
Source A: category map
Section titled “Source A: category map”This source is structured and always on. The classifier put the question in
triggered_abilities. The YAML says that category maps to CR sections 603 and 113.3. Every
rule in those sections goes into the context. Rules that share a word with the question
come first, ranked by the same relevance score as source B. The secondary categories’ rules
are added the same way, after the other sources.
It costs nothing and never misses when the classifier is right. It gives the model the surrounding rules it needs even when the “obvious” rule alone is not enough. It fails when the classifier is wrong or when the answer lives in a section nobody would file the question under.
Source B: full-text search
Section titled “Source B: full-text search”This source matches keywords, using Postgres’s built-in full-text engine. Each rule row has a
generated tsvector column: the text split into words, lower-cased, reduced to word stems
(“triggers” → “trigger”), with common words like “the” removed. The question is reduced the
same way, and a row matches if it contains any of those stems. Rows are ranked with
ts_rank_cd, a relevance score in the same family as BM25. Such scores count the matching
words, weight rare words higher, and discount long documents. Matches on concept phrases
from the extraction weigh double those on the raw question. The top 12 rows are taken.
This finds rules that share vocabulary with the question: “leaves the battlefield”, “in response”, “upkeep”. It is exact, cheap, and needs no external service. It fails when the user and the CR use different words for the same idea (“dies” versus “is put into a graveyard from the battlefield”). Common words also distract it.
Source C: vector similarity
Section titled “Source C: vector similarity”This source matches meaning.
Embeddings. An embedding model is a neural network that turns a piece of text into a
list of numbers, a vector, typically 512 to 3072 floats long. This bot uses Voyage AI’s
voyage-3.5 at 1024 dimensions by default. The model is trained so that texts with similar
meaning land near each other in that space, whatever words they use. “Dies in response
to its trigger” and “an ability that triggers on leaving the battlefield resolves even if
the source is gone” should be neighbours even though they share almost no words.
Similarity. Two vectors are compared by cosine similarity: the cosine of the angle
between them, 1.0 for identical direction, 0 for unrelated. pgvector exposes this as the
<=> operator (cosine distance, 1 minus similarity, so smaller is closer). The source is
essentially:
SELECT ... FROM rulesWHERE parent_id IS NULL AND embedding IS NOT NULLORDER BY embedding <=> $question_vectorLIMIT 12(The real query also restricts id to the NNN.N rule pattern.)
Indexing. Comparing the question against 2,000 rule vectors by brute force would be
fine at this scale. pgvector also provides an HNSW index (Hierarchical Navigable
Small World), a graph structure that finds approximate nearest neighbours quickly. It is
approximate, so it can occasionally miss the true nearest row. That is acceptable here
because the other two sources cover for it. The index is partial: it covers rule-level rows
only (WHERE parent_id IS NULL). Every row it returns is therefore one the query wants, and
no filter applied afterwards can shrink the result below the limit.
Query versus document. Voyage’s API takes an input_type of document or query.
The model embeds a short question differently from a long passage so that the two match
up better. The ingest job embeds rules as documents. The retriever embeds the user’s
question as a query.
Cost and storage. Embedding is paid per token, once per rule, at ingest time. A new CR
release re-embeds only the rules whose text changed. The loader nulls those embeddings and
the next refresh’s embed step fills them. Each question costs one small embedding call.
The vector search has failures of its own. It matches by meaning, so it returns rules that
are about the same theme without being the one that decides the question. It cannot tell
702.19 from 702.20 if their wording is similar. Its operational problems are listed in
§7.
Source order
Section titled “Source order”The sources are merged in priority order, and a rule found twice is kept once:
- the primary category’s rules that share a word with the question,
- full-text hits,
- vector hits,
- the rest of the primary category,
- each secondary category.
The synthesis budget shows the model only the front of this list, so the order decides what
it reads. The primary category’s matching rules always survive, and the secondary
categories are trimmed first. The retrieval gate in the eval suite (judge-eval recall) requires that at least
90% of the gold set’s expected rule ids appear in the context, and at least 75% in the part
the budget shows.
Whatever all three miss is left to the one lookup_rules tool round in synthesis. Having
read the material, the model can ask for rules by number once.
Prior calls
Section titled “Prior calls”The prior-call query is the feedback loop. Earlier answers are stored with their own
embedding (of the question). The prior-call query picks calls in the same categories, about at least one
of the same cards, not retired and not down-voted. It orders them by vector distance to the
new question. The rating is a Bayesian-smoothed mean of the Discord ratings (an
anonymous web rating is never counted): (2.0 × 3 + Σ scores) / (3 + n).
That is “pretend there were three votes of 2.0 before anyone voted”. One 3 does not make a
call look perfect, and one 1 does not bury it. The latest rating from someone with the
Judge role overrides the crowd. Calls scoring under 1.5 are excluded: with five or more
votes on the crowd’s score, at once on a judge’s.
Prior calls are rendered after all CR material, as examples of how a question was answered, and the prompt says they never outrank it.
6. Synthesis guardrails
Section titled “6. Synthesis guardrails”Structured output and one tool round
Section titled “Structured output and one tool round”The model does not write free text for the program to parse. It fills a JSON schema derived
from the Verdict struct. The lookup_rules tool round is bounded to one by a
typestate:
Synth<Fresh>can send and becomeSynth<ToolRequested>.Synth<ToolRequested>can fulfil the request and becomeSynth<Final>.Synth<Final>has no method that requests tools.
A second tool round is therefore code that does not compile.
The model can still ask for a second round, or send ids that are not rule ids. Neither
is fetched. Each is a typed rejection (ToolMisuse) and gets the one retry, with a notice
saying what the call allows. The retry may make the call itself only when the first one
could not be read, because then nothing was fetched.
Citation validation
Section titled “Citation validation”The validator (crates/core/src/verdict.rs) checks each citation against the context:
- The reference must exist: the rule id was shown or fetched, the ruling key was rendered under that card, the prior-call id was in the list, the Oracle face exists.
- The quote must be a contiguous substring of that source’s text.
The most common rejection in practice was punctuation. The CR is typeset with curly
apostrophes (doesn’t) and em dashes. Models reliably retype them as ASCII (doesn't)
even when told not to. Rejecting a correct citation over one character wastes the only
retry. So the comparison (crates/core/src/quote.rs) maps each character to one plain
equivalent: curly quotes to straight, every dash to a hyphen, a non-breaking space to a
space. The leniency applies only while matching. What gets stored is the source’s span,
not the model’s string, so the database is byte-exact and later strict checks stay strict.
Case, word order and line breaks must still match, so a paraphrase is still rejected.
A verdict with no citations, or an answer under 40 characters, is rejected as empty. Two
fields the model could misreport, source and cr_version, are not in the model’s
schema. The program stamps them from the extraction and the retrieved chunks.
The validated type
Section titled “The validated type”Verdict<Unvalidated> is what JSON decodes into. Verdict<Validated> is the only type
CallStore::persist and the Discord renderer accept, and the only way to make one is
validate(). Serde’s Deserialize is implemented for the unvalidated state only, so
decoding straight into the validated one does not compile, and deleting the check is a
type error.
The retry
Section titled “The retry”There is one retry, with a “Previous attempt rejected” notice showing the failing citation and why. The retry starts with the tool disabled, so the model cannot spend another round. Any rules the first attempt fetched are rendered regardless of budget. The first rejection is logged at INFO so that when the second attempt also fails, the operator can read both.
MTG Judgebot is unofficial Fan Content permitted under the Fan Content Policy. Not approved or endorsed by Wizards of the Coast. Portions of the materials used are property of Wizards of the Coast. ©Wizards of the Coast LLC.
The Comprehensive Rules come from Wizards of the Coast. Card data, rulings and card symbols come from Scryfall, which is not affiliated with this project. Rule links go to the Yawgatog mirror. License and attribution.