Explainer 4: Providers, evaluation, technology

The last part of the explainer.

11. Providers

Which chat model and which embedder to use is a judge.toml file, not code. The pipeline talks to a provider-neutral ChatRequest/ChatResponse in crates/llm. crates/anthropic and crates/openai are backends that own their wire formats. Anthropic can be reached directly, through a proxy, or on AWS and GCP with the platform’s own credential chains. Any OpenAI-compatible chat-completions server works, which covers local models. Embeddings come from Voyage or any OpenAI-compatible /embeddings endpoint.

Where a backend cannot enforce the output schema server-side, the adapter appends the schema to the user turn. The system prompt is therefore byte-identical on every backend, and a pinned digest and golden fixtures guard that. Every model on a paid provider must have a price so the cap can reserve for it. Unknown Anthropic models fall back to a table that errs high.

---

12. Evaluation

eval/gold.yaml holds 21 adversarially verified questions with the rule ids an answer must cite, plus per-question lists of equivalent ids that state the same fact. Two gates:

• judge-eval recall runs only extraction, resolution and retrieval and fails below 90% of expected rule ids in context. No model spend for synthesis.
• judge-eval answer runs the full pipeline and scores the answers. Runs are stored and can be re-scored for free after the gold set is edited.

The gold set is extended whenever capability is added. It is the closest thing the system has to a regression suite for the probabilistic parts.

---

13. Technology choices

Rust. Chosen for what the compiler enforces (docs/DECISIONS.md D1 lists nine invariants, D2 the languages it was weighed against). In this codebase that means:

• Exhaustive enums: every consumer of Resolution, Citation and JudgeError handles every case, so “ambiguous” cannot be silently treated as “resolved”.
• Newtypes with validators (nutype): a RuleId matches the CR’s id pattern, a rating is 1 to 3, a NonEmpty<Face> list cannot be empty. Invalid data is unconstructible.
• Typestates: Verdict<Validated> and Synth<Final> make “unvalidated answer reaches Discord” and “second tool round” compile errors.
• Result everywhere and lints that deny unwrap, expect, panic and slice indexing, so every failure on the judge path is a value the Discord layer must render.
• Compile-time checked SQL (sqlx): every query is checked against the schema at build time, including pgvector columns, so a renamed column is a build failure.
• A crate graph as an effect fence: crates/core has no I/O dependencies, so pure logic (resolution rules, context assembly, validation) cannot sneak in a network call.

The cost accepted: the HTTP clients for the model APIs are hand-written and pinned against golden request fixtures.

Postgres 16 + pgvector + pg_trgm. One database does relational storage, full-text search, trigram fuzzy matching and vector search, all joinable in one query with transactions and advisory locks across them. At this scale a separate vector database would add an operational component and a consistency problem for nothing. HNSW indexes make approximate nearest neighbour search fast on a few thousand rows.

Voyage AI embeddings (voyage-3.5, 1024 dimensions, by default). A query/document distinction and a hosted API, so the zero-config setup does not need a local model (which is not worth the trouble under WSL2). Swappable by config.

Anthropic Claude for both model stages. A low-effort call for extraction and a high-effort call with tool use and structured outputs for synthesis. Prompt caching matters for the large synthesis turn. The provider seam means this is a default, not a lock-in.

serenity + poise for Discord, axum for HTTP, rmcp for MCP, SolidJS + Vite for the page. All conventional, well-maintained choices for their niches.

Docker Compose behind a Cloudflare Tunnel. One host, no open inbound ports, a CI-built image, nightly data refresh as a cron job rather than a service, weekly backups to R2. docs/DEPLOYMENT.md is the runbook.

---

14. Glossary

• RAG (retrieval-augmented generation): fetch relevant documents first, then have a model answer from them rather than from memory.
• Embedding: a fixed-length vector of floats that an embedding model produces from text, such that similar meanings give nearby vectors.
• Vector space / embedding space: the set of vectors one specific model produces. Vectors from different models, or the same model at a different width, are not comparable. This bot records which space the database holds.
• Cosine similarity / distance: how aligned two vectors are. pgvector’s <=> is the distance (0 is identical).
• HNSW: a graph index for approximate nearest-neighbour search. It is fast and occasionally misses the true nearest.
• Dimensions: the length of the vector (1024 here). More is not automatically better, because it costs storage and index time.
• Full-text search / tsvector / BM25: keyword search with stemming and rarity weighting. Postgres’s ts_rank_cd is its relevance scorer.
• Trigram (pg_trgm): similarity based on shared three-character windows. Good for typos in names.
• Hybrid retrieval: combining keyword, semantic and structured search because each fails differently.
• Structured output: the model API enforces a JSON schema on the reply.
• Tool use / function calling: the model asks the program to run a named function (lookup_rules) and gets the result back before answering.
• Prompt caching: the provider caches a stable prefix of the prompt and charges much less to reread it.
• Typestate: encoding an object’s lifecycle stage in its type so that the wrong operation at the wrong stage does not compile.
• Oracle text: a Magic card’s current official wording, as opposed to what is printed.
• CR: the Comprehensive Rules. MTR / IPG: tournament policy documents, out of scope.

15. Where to go next

• The pipeline: crates/core/src/judge.rs, then verdict.rs and quote.rs.
• Retrieval SQL: crates/bot/src/db/retrieve.rs and rules.rs.
• The resolution ladder: crates/bot/src/db/resolve.rs (its module comment is thorough).
• Vector-space bookkeeping: crates/bot/src/db/space.rs, crates/embed/src/space.rs.
• The model contract: crates/bot/src/prompts/synth_system.md.
• The spend cap: crates/llm/src/spend.rs.
• The reference and the reasoning: docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/PROVIDERS.md.
• Outside reading: the pgvector README (HNSW, distance operators), the Postgres full-text search chapter, Voyage AI’s docs on input_type, and Anthropic’s docs on structured outputs, tool use and prompt caching.