Explainer 2: Data and retrieval

Part two of the explainer. Part one covered the pipeline end to end.

4. The data and where it comes from

Everything lives in one Postgres 16 database with two extensions: pgvector (vector columns and indexes) and pg_trgm (trigram similarity). Migrations are in crates/bot/migrations/.

Table	Source	Refreshed	Notes
cards, card_faces	Scryfall bulk oracle_cards	nightly	one row per Oracle identity, and faces hold the Oracle text
printed_names	Scryfall bulk default_cards	nightly	every name ever printed, for old or errata’d names
rulings	Scryfall bulk rulings	nightly	keyed by a hash of the content, so a re-import is the same ruling
rules, glossary	the CR .txt from Wizards	on release, detected nightly	see chunking below
card_aliases	data/aliases.yaml	when edited	nicknames
card_notes	data/notes.yaml	when edited	nightmare cards
categories	data/categories.yaml, via the compiled enum	with each CR load	category → CR subsections
calls, ratings	the bot	continuous	answers and their votes
embedding_space	ingest embed	on switch	one row: which embedding model the vectors came from
agent_sessions	the agent surface	continuous	state for the step-by-step agent mode

Scale is small: ~30k cards, ~2k rule chunks, well under 10k calls. This matters for technology choices. Nothing here needs a dedicated vector database.

How the CR is chunked

A text file is useless to search until it is cut into pieces. The cut is the most consequential decision in any RAG system, because a piece is what gets found, what gets shown, and what gets cited.

The CR is a numbered hierarchy: section 702 (Keyword Abilities) → rule 702.19 (Trample) → sub-rules 702.19a, 702.19b. The parser emits rows at two granularities:

• Rule-level rows (702.19): the body is the rule’s own sentence plus every lettered sub-rule and every Example: paragraph beneath it. These are the search unit. They get embeddings and appear in retrieval results. A rule plus its sub-rules is usually one coherent idea of a few hundred words, the right size for a model to read in one piece.
• Leaf rows (702.19b): one line each, with parent_id = 702.19. These are the citation unit. A model that quotes sub-rule b should cite 702.19b, not the rule above it.

Scoring and lookups treat a leaf and its parent as covering each other. No rows exist for three-digit sections. Asking for 702 expands to every rule in it.

---

5. Retrieval

This section holds most of the vector-database material. The bot combines three search techniques because each fails differently.

Leg A: category map

This leg is structured and always on. The classifier put the question in triggered_abilities. The YAML says that category maps to CR sections 603 and 113.3. Every rule in those sections goes into the context.

This is dumb and reliable. It costs nothing and never misses when the classifier is right. It gives the model the surrounding rules it needs even when the “obvious” rule alone is not enough. It fails when the classifier is wrong or when the answer lives in a section nobody would file the question under.

Leg B: full-text search

This leg matches keywords. Postgres has a built-in full-text engine. Each rule row has a generated tsvector column: the text tokenised, lower-cased, stemmed (“triggers” → “trigger”) and stop-words removed. The query is turned into the same lexemes, OR-ed together. Rows are ranked with ts_rank_cd, a relevance score in the same family as BM25 (frequency of matching terms, weighted by how rare they are, discounted by document length). Concept phrases from the extraction count double against the raw question. The top 12 rows are taken.

This finds rules that share vocabulary with the question: “leaves the battlefield”, “in response”, “upkeep”. It is exact, cheap, and needs no external service. It fails when the user and the CR use different words for the same idea (“dies” versus “is put into a graveyard from the battlefield”). Common words also distract it.

Leg C: vector similarity

This leg matches meaning. It is the piece most people are new to, so it gets the most detail.

Embeddings. An embedding model is a neural network that turns a piece of text into a list of numbers, a vector, typically 512 to 3072 floats long. This bot uses Voyage AI’s voyage-3.5 at 1024 dimensions by default. The model is trained so that texts with similar meaning land near each other in that space, whatever words they use. “Dies in response to its trigger” and “an ability that triggers on leaving the battlefield resolves even if the source is gone” should be neighbours even though they share almost no words.

Similarity. Two vectors are compared by cosine similarity: the cosine of the angle between them, 1.0 for identical direction, 0 for unrelated. pgvector exposes this as the <=> operator (cosine distance, 1 minus similarity, so smaller is closer). The leg is essentially:

SELECT ... FROM rules
WHERE parent_id IS NULL AND embedding IS NOT NULL
ORDER BY embedding <=> $question_vector
LIMIT 12

(The real query also restricts id to the NNN.N rule pattern.)

Indexing. Comparing the question against 2,000 rule vectors by brute force would be fine at this scale. pgvector also provides an HNSW index (Hierarchical Navigable Small World), a graph structure that finds approximate nearest neighbours quickly. It is approximate, so it can occasionally miss the true nearest row. That is acceptable here because the union with the other two legs covers for it. Because the index is partial, over rule-level rows only (WHERE parent_id IS NULL), every candidate it yields is usable. A post-filter cannot shrink the result below the limit.

Query versus document. Voyage’s API takes an input_type of document or query. The model embeds a short question differently from a long passage so that the two match up better. The ingest job embeds rules as documents. The retriever embeds the user’s question as a query.

Cost and storage. Embedding is paid per token, once per rule, at ingest time. A new CR release re-embeds only the rules whose text changed. The loader nulls those embeddings and the nightly ingest embed fills them. Each question costs one small embedding call.

The vector leg’s failures are instructive too. It is fuzzy by design, so it returns rules that are about the same theme without being the one that decides the question. It cannot tell 702.19 from 702.20 if their wording is similar. It also has a class of operational problems that get their own section (§7).

Leg order

The legs are unioned in priority order and deduplicated by rule id: category map first, then full-text, then vector. When the budget cuts, chunks are kept in that order, so the structured leg survives and the fuzziest leg is trimmed first. The retrieval gate in the eval suite (judge-eval recall) requires that at least 90% of the gold set’s expected rule ids appear in the context.

The safety net for whatever all three miss is the one lookup_rules tool round in synthesis. Having read the material, the model can ask for rules by number once.

Prior calls

The prior-call query is the feedback loop. Earlier answers are stored with their own embedding (of the question). The leg picks calls in the same categories, about at least one of the same cards, not retired and not down-voted. It orders them by vector distance to the new question. The rating is a Bayesian-smoothed mean: (2.0 × 3 + Σ scores) / (3 + n). That is “pretend there were three votes of 2.0 before anyone voted”. One 3 does not make a call look perfect, and one 1 does not bury it. The latest rating from someone with the Judge role overrides the crowd. Calls scoring under 1.5 with five or more votes are excluded.

Prior calls are rendered after all CR material, and the prompt says they never outrank it. They are examples of how a question was answered, not authorities.

---

6. Synthesis guardrails

Structured output and one tool round

The model does not write free text for the program to parse. It fills a JSON schema derived from the Verdict struct. The lookup_rules tool round is bounded to one by a typestate:

• Synth<Fresh> can send and become Synth<ToolRequested>.
• Synth<ToolRequested> can fulfil the request and become Synth<Final>.
• Synth<Final> has no method that requests tools.

A runaway loop is not a bug for a test to catch. It is code that does not compile.

Citation validation

The validator (crates/core/src/verdict.rs) checks each citation against the context:

• The reference must exist: the rule id was shown or fetched, the ruling key was rendered under that card, the prior-call id was in the list, the Oracle face exists.
• The quote must be a contiguous substring of that source’s text.

The most common rejection in practice was punctuation. The CR is typeset with curly apostrophes (doesn’t) and em dashes. Models reliably retype them as ASCII (doesn't) even when told not to. Rejecting a correct citation over one character wastes the only retry. So the comparison (crates/core/src/quote.rs) canonicalises each character, one char to one char: curly to straight, every dash to a hyphen, non-breaking space to space. It then stores the source’s span, not the model’s. The leniency applies at match time only. What lands in the database is byte-exact, so later strict checks stay strict. Case, word order and line breaks must still match, so a paraphrase is still rejected.

A verdict with no citations, or an answer under 40 characters, is rejected as empty. Two fields the model might be tempted to lie about, source and cr_version, are not in the model’s schema. The program stamps them from the extraction and the retrieved chunks.

The validated type

Verdict<Unvalidated> is what JSON decodes into. Verdict<Validated> is the only type CallStore::persist and the Discord renderer accept, and the only way to make one is validate(). Serde’s Deserialize is implemented for the unvalidated state only, so decoding straight into the validated one does not compile. Deleting the check is a type error, not a silent regression.

The retry

There is one retry, with a “Previous attempt rejected” notice showing the failing citation and why. The retry starts with the tool disabled, so the model cannot spend another round. Any rules the first attempt fetched are rendered regardless of budget. The first rejection is logged at INFO so that when the second attempt also fails, the operator can read both.

---