Explainer 4: Providers, evaluation, technology
The last part of the explainer.
11. Providers
Section titled “11. Providers”Which chat model and which embedder to use is a judge.toml file, not code. The pipeline
talks to a provider-neutral ChatRequest/ChatResponse in crates/llm. crates/anthropic
and crates/openai are backends that own their wire formats. Anthropic can be reached
directly, through a proxy, or on AWS and GCP with the platform’s own credential chains. Any
OpenAI-compatible chat-completions server works, which covers local models. Embeddings come
from Voyage or any OpenAI-compatible /embeddings endpoint.
Where a backend cannot enforce the output schema server-side, the adapter appends the schema to the user turn. The system prompt is therefore byte-identical on every backend, and a pinned digest and golden fixtures guard that. Every model on a paid provider must have a price so the cap can reserve for it. Unknown Anthropic models fall back to a table that errs high.
12. Evaluation
Section titled “12. Evaluation”eval/gold.yaml holds 24 questions: 21 adversarially verified, one a human correction of
a reported wrong answer and two about unreleased cards (D42). Each lists the rule ids a correct answer must cite
(decisive) and background rule ids it may leave out (supporting), plus equivalent ids that
state the same fact. Two gates:
judge-eval recallruns card resolution and retrieval, taking the extraction from the gold file instead of a model call. It fails below 90% of expected rule ids (decisive and supporting) in context, or below 75% in the part the budget shows. It makes no chat-model calls.judge-eval answerruns the full pipeline and scores citation recall on the decisive ids, reporting the supporting ids that were cited as well. Each row also records why the first attempt was rejected, when the retry ran, and how many stub citations were dropped. Runs are stored and can be re-scored for free after the gold set is edited.
Citation recall cannot tell a right answer from a wrong one that cites the right rules.
judge-eval grade is a third measure, a score rather than a gate: a model reads each
stored answer with its quotes and the reference, and says whether the ruling follows from
the quotes, whether it agrees with the reference, and which remarks are wrong
(docs/DECISIONS.md D28).
The gold set is extended whenever capability is added. It is the closest thing the system has to a regression suite for the probabilistic parts.
13. Technology choices
Section titled “13. Technology choices”Rust. Chosen for what the compiler enforces (docs/DECISIONS.md D1 lists nine
invariants, D2 the languages it was weighed against). In this codebase that means:
- Exhaustive enums: every consumer of
Resolution,CitationandJudgeErrorhandles every case, so “ambiguous” cannot be silently treated as “resolved”. - Newtypes with validators (
nutype): aRuleIdmatches the CR’s id pattern, a rating is 1 to 3, aNonEmpty<Face>list cannot be empty. Invalid data is unconstructible. - Typestates:
Verdict<Validated>andSynth<Final>make “unvalidated answer reaches Discord” and “second tool round” compile errors. Resulteverywhere and lints that denyunwrap,expect,panicand slice indexing, so every failure on the judge path is a value the Discord layer must render.- Compile-time checked SQL (
sqlx): every query is checked against the schema at build time, including pgvector columns, so a renamed column is a build failure. - A crate graph as an effect fence:
crates/corehas no I/O dependencies, so pure logic (resolution rules, context assembly, validation) cannot make a network call.
The cost accepted: the HTTP clients for the model APIs are hand-written and pinned against golden request fixtures.
Postgres 16 + pgvector + pg_trgm. One database does relational storage, full-text search, trigram fuzzy matching and vector search, all joinable in one query with transactions and advisory locks across them. At this scale a separate vector database would add an operational component and a consistency problem for nothing. HNSW indexes make approximate nearest neighbour search fast on a few thousand rows.
Voyage AI embeddings (voyage-3.5, 1024 dimensions, by default). A query/document
distinction and a hosted API, so the zero-config setup does not need a local model (which
is not worth the trouble under WSL2). Swappable by config.
Anthropic Claude for both model stages. Opus 5.5 at medium effort for both: a short
call for extraction and a call with tool use and structured outputs for synthesis. On the
gold set, medium extraction never wrote a card’s name in part, which low occasionally
did, and medium synthesis agreed with the reference as often as high, for less. A model
without a measurement extracts at low and synthesizes at high. Prompt caching
matters for the large synthesis turn. This is a default that judge.toml can change
(§11).
axum for HTTP, rmcp for MCP, SolidJS + Vite for the web app: conventional, well-maintained choices for their niches. Discord has no library: the bot uses a small part of its API (zero gateway intents, two events, nine REST calls), so it writes those itself (D43).
Docker Compose on one host. A CI-built image run as one judgebot service whose
roles are launch options (D25), reached through a bundled Caddy: on the host’s loopback
by default, at a public HTTPS name with a Let’s Encrypt certificate under the public
profile, or through a Cloudflare Tunnel with no open inbound port (D41). It has a
daily data refresh that the running process schedules in the database rather than a
host cron job (D24), weekly backups to any S3-compatible store from an opt-in container
that keeps its schedule in the bucket (D26). docs/DEPLOYMENT.md is the runbook.
14. Glossary
Section titled “14. Glossary”- RAG (retrieval-augmented generation): fetch relevant documents first, then have a model answer from them rather than from memory.
- Embedding: a fixed-length vector of floats that an embedding model produces from text, such that similar meanings give nearby vectors.
- Vector space / embedding space: the set of vectors one specific model produces. Vectors from different models, or the same model at a different width, are not comparable. This bot records which space the database holds.
- Cosine similarity / distance: how aligned two vectors are. pgvector’s
<=>is the distance (0 is identical). - HNSW: a graph index for approximate nearest-neighbour search. It is fast and occasionally misses the true nearest.
- Dimensions: the length of the vector (1024 here). More is not automatically better, because it costs storage and index time.
- Full-text search / tsvector / BM25: keyword search with stemming and rarity
weighting. Postgres’s
ts_rank_cdis its relevance scorer. - Trigram (pg_trgm): similarity based on shared three-character windows. Good for typos in names.
- Hybrid retrieval: combining keyword, semantic and structured search because each fails differently.
- Structured output: the model API enforces a JSON schema on the reply.
- Tool use / function calling: the model asks the program to run a named function
(
lookup_rules) and gets the result back before answering. - Prompt caching: the provider caches a stable prefix of the prompt and charges much less to reread it.
- Typestate: encoding an object’s lifecycle stage in its type so that the wrong operation at the wrong stage does not compile.
- Oracle text: a Magic card’s current official wording, as opposed to what is printed.
- CR: the Comprehensive Rules. MTR / IPG: tournament policy documents, out of scope.
15. Next
Section titled “15. Next”- The pipeline:
crates/core/src/judge.rs, thenverdict.rsandquote.rs. - Retrieval SQL:
crates/bot/src/db/retrieve.rsandrules.rs. - The resolution order:
crates/bot/src/db/resolve.rs(its module comment is thorough). - Vector-space bookkeeping:
crates/bot/src/db/space.rs,crates/embed/src/space.rs. - The model contract:
crates/bot/src/prompts/synth_system.md, andextract_system.mdbeside it for the extraction stage. - The spend cap:
crates/llm/src/spend.rs. - The reference and the reasoning:
docs/ARCHITECTURE.md,docs/DECISIONS.md,docs/PROVIDERS.md. - Outside reading: the pgvector README (HNSW, distance operators), the Postgres full-text
search chapter, Voyage AI’s docs on
input_type, and Anthropic’s docs on structured outputs, tool use and prompt caching.
MTG Judgebot is unofficial Fan Content permitted under the Fan Content Policy. Not approved or endorsed by Wizards of the Coast. Portions of the materials used are property of Wizards of the Coast. ©Wizards of the Coast LLC.
The Comprehensive Rules come from Wizards of the Coast. Card data, rulings and card symbols come from Scryfall, which is not affiliated with this project. Rule links go to the Yawgatog mirror. License and attribution.