Skip to content

Explainer 4: Providers, evaluation, technology

The last part of the explainer.

Which chat model and which embedder to use is a judge.toml file, not code. The pipeline talks to a provider-neutral ChatRequest/ChatResponse in crates/llm. crates/anthropic and crates/openai are backends that own their wire formats. Anthropic can be reached directly, through a proxy, or on AWS and GCP with the platform’s own credential chains. Any OpenAI-compatible chat-completions server works, which covers local models. Embeddings come from Voyage or any OpenAI-compatible /embeddings endpoint.

Where a backend cannot enforce the output schema server-side, the adapter appends the schema to the user turn. The system prompt is therefore byte-identical on every backend, and a pinned digest and golden fixtures guard that. Every model on a paid provider must have a price so the cap can reserve for it. Unknown Anthropic models fall back to a table that errs high.


eval/gold.yaml holds 24 questions: 21 adversarially verified, one a human correction of a reported wrong answer and two about unreleased cards (D42). Each lists the rule ids a correct answer must cite (decisive) and background rule ids it may leave out (supporting), plus equivalent ids that state the same fact. Two gates:

  • judge-eval recall runs card resolution and retrieval, taking the extraction from the gold file instead of a model call. It fails below 90% of expected rule ids (decisive and supporting) in context, or below 75% in the part the budget shows. It makes no chat-model calls.
  • judge-eval answer runs the full pipeline and scores citation recall on the decisive ids, reporting the supporting ids that were cited as well. Each row also records why the first attempt was rejected, when the retry ran, and how many stub citations were dropped. Runs are stored and can be re-scored for free after the gold set is edited.

Citation recall cannot tell a right answer from a wrong one that cites the right rules. judge-eval grade is a third measure, a score rather than a gate: a model reads each stored answer with its quotes and the reference, and says whether the ruling follows from the quotes, whether it agrees with the reference, and which remarks are wrong (docs/DECISIONS.md D28).

The gold set is extended whenever capability is added. It is the closest thing the system has to a regression suite for the probabilistic parts.


Rust. Chosen for what the compiler enforces (docs/DECISIONS.md D1 lists nine invariants, D2 the languages it was weighed against). In this codebase that means:

  • Exhaustive enums: every consumer of Resolution, Citation and JudgeError handles every case, so “ambiguous” cannot be silently treated as “resolved”.
  • Newtypes with validators (nutype): a RuleId matches the CR’s id pattern, a rating is 1 to 3, a NonEmpty<Face> list cannot be empty. Invalid data is unconstructible.
  • Typestates: Verdict<Validated> and Synth<Final> make “unvalidated answer reaches Discord” and “second tool round” compile errors.
  • Result everywhere and lints that deny unwrap, expect, panic and slice indexing, so every failure on the judge path is a value the Discord layer must render.
  • Compile-time checked SQL (sqlx): every query is checked against the schema at build time, including pgvector columns, so a renamed column is a build failure.
  • A crate graph as an effect fence: crates/core has no I/O dependencies, so pure logic (resolution rules, context assembly, validation) cannot make a network call.

The cost accepted: the HTTP clients for the model APIs are hand-written and pinned against golden request fixtures.

Postgres 16 + pgvector + pg_trgm. One database does relational storage, full-text search, trigram fuzzy matching and vector search, all joinable in one query with transactions and advisory locks across them. At this scale a separate vector database would add an operational component and a consistency problem for nothing. HNSW indexes make approximate nearest neighbour search fast on a few thousand rows.

Voyage AI embeddings (voyage-3.5, 1024 dimensions, by default). A query/document distinction and a hosted API, so the zero-config setup does not need a local model (which is not worth the trouble under WSL2). Swappable by config.

Anthropic Claude for both model stages. Opus 5.5 at medium effort for both: a short call for extraction and a call with tool use and structured outputs for synthesis. On the gold set, medium extraction never wrote a card’s name in part, which low occasionally did, and medium synthesis agreed with the reference as often as high, for less. A model without a measurement extracts at low and synthesizes at high. Prompt caching matters for the large synthesis turn. This is a default that judge.toml can change (§11).

axum for HTTP, rmcp for MCP, SolidJS + Vite for the web app: conventional, well-maintained choices for their niches. Discord has no library: the bot uses a small part of its API (zero gateway intents, two events, nine REST calls), so it writes those itself (D43).

Docker Compose on one host. A CI-built image run as one judgebot service whose roles are launch options (D25), reached through a bundled Caddy: on the host’s loopback by default, at a public HTTPS name with a Let’s Encrypt certificate under the public profile, or through a Cloudflare Tunnel with no open inbound port (D41). It has a daily data refresh that the running process schedules in the database rather than a host cron job (D24), weekly backups to any S3-compatible store from an opt-in container that keeps its schedule in the bucket (D26). docs/DEPLOYMENT.md is the runbook.


  • RAG (retrieval-augmented generation): fetch relevant documents first, then have a model answer from them rather than from memory.
  • Embedding: a fixed-length vector of floats that an embedding model produces from text, such that similar meanings give nearby vectors.
  • Vector space / embedding space: the set of vectors one specific model produces. Vectors from different models, or the same model at a different width, are not comparable. This bot records which space the database holds.
  • Cosine similarity / distance: how aligned two vectors are. pgvector’s <=> is the distance (0 is identical).
  • HNSW: a graph index for approximate nearest-neighbour search. It is fast and occasionally misses the true nearest.
  • Dimensions: the length of the vector (1024 here). More is not automatically better, because it costs storage and index time.
  • Full-text search / tsvector / BM25: keyword search with stemming and rarity weighting. Postgres’s ts_rank_cd is its relevance scorer.
  • Trigram (pg_trgm): similarity based on shared three-character windows. Good for typos in names.
  • Hybrid retrieval: combining keyword, semantic and structured search because each fails differently.
  • Structured output: the model API enforces a JSON schema on the reply.
  • Tool use / function calling: the model asks the program to run a named function (lookup_rules) and gets the result back before answering.
  • Prompt caching: the provider caches a stable prefix of the prompt and charges much less to reread it.
  • Typestate: encoding an object’s lifecycle stage in its type so that the wrong operation at the wrong stage does not compile.
  • Oracle text: a Magic card’s current official wording, as opposed to what is printed.
  • CR: the Comprehensive Rules. MTR / IPG: tournament policy documents, out of scope.
  • The pipeline: crates/core/src/judge.rs, then verdict.rs and quote.rs.
  • Retrieval SQL: crates/bot/src/db/retrieve.rs and rules.rs.
  • The resolution order: crates/bot/src/db/resolve.rs (its module comment is thorough).
  • Vector-space bookkeeping: crates/bot/src/db/space.rs, crates/embed/src/space.rs.
  • The model contract: crates/bot/src/prompts/synth_system.md, and extract_system.md beside it for the extraction stage.
  • The spend cap: crates/llm/src/spend.rs.
  • The reference and the reasoning: docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/PROVIDERS.md.
  • Outside reading: the pgvector README (HNSW, distance operators), the Postgres full-text search chapter, Voyage AI’s docs on input_type, and Anthropic’s docs on structured outputs, tool use and prompt caching.