Skip to content

Architecture

The design reference, kept current with the code. It was written stack-independent, before the language was chosen (Rust, 2026-08-29). docs/DECISIONS.md records why each choice was made. docs/EXPLAINER.md is the narrative version for someone new to the ideas.

A Discord bot that answers Magic: The Gathering rules questions. Users reference cards by full name, partial name, or nickname (“bob” → Dark Confidant). Every answer cites its sources: Comprehensive Rules (CR) sections, Scryfall rulings, and/or prior rated calls. Answers and user-submitted calls are rated 1–3 (incorrect / partially correct / correct), the two that find fault with a reason for the operator’s feedback queue. Ratings feed back into retrieval as examples and never outrank the CR; reasons never reach the model.

Rules questions have the shape “card A + card B + rule concept C”, and the design follows from that:

  • Card names are entities → extraction + lookup, not embeddings.
  • CR is a numbered hierarchy → chunk at the rule level (e.g. 702.19 with all lettered sub-rules and Example: lines), keep parent links, and map question categories to whole subsections (e.g. 613, 614).
  • Scryfall rulings are keyed by card → fetch directly once cards resolve.
  • Three retrieval sources for the fuzzy part: category map (structured), BM25/full-text (keywords like “leaves the battlefield”), vector (semantic).
  • The CR always outranks prior calls.
Discord message (+ last N Q&A in the same thread)
│
▼
[1] Entity extraction (LLM, the model's measured effort, structured output)
Splits the message into card-name spans and rules concepts. Runs FIRST so
fuzzy matching sees only candidate spans, not rules vocabulary.
Model choice: `judge_bot::config` picks a provider per stage, from a
`judge.toml`, else Anthropic direct from the environment. A provider is
either Anthropic's Messages API or any OpenAI-compatible chat completions
server. The Messages API is reached direct, through a proxy, or on a
cloud account: Claude Platform on AWS and Bedrock (SigV4), Vertex AI
(ADC). Cloud credentials come from the platform chain and are probed at
startup.
Every model sits behind the process's one spend-capped `Metered`, and
the embedder behind a `MeteredEmbedder` on the same meter.
A backend reports its `Capabilities`. If it cannot enforce the output
schema server-side, the adapter appends the schema to the *user turn*.
The system prompt is pinned by digest and is byte-identical on every
backend.
│
▼
[2] Card resolution (per span)
A typed resolution order, each step tried only when the one above found nothing:
alias → possessive-stripped alias ("bob's" → "bob") →
exact name → printed-name table → short name before the comma ("Ragavan")
→ the name after the comma, a leading "the" dropped, two words or more
("Nimble Pilferer"; a one-word "ravager" goes on to fuzzy, since
players mean other cards by it). When another card's name holds the
same words ("Grand Arbiter Augustin IV" for Dovin, Grand Arbiter's
"Grand Arbiter"), every such card is offered instead: Ambiguous, never
resolved → alias as a suffix → trigram fuzzy
A [[bracketed]] span has its own resolution order: exact name → printed name →
alias (whole span). It is CardSpan::Exact, chosen by an exhaustive
match, so no looser step can resolve it. An alias maps one spelling to
one card, so [[bob]] resolves. A miss is never resolved, only offered
as Ambiguous:
- the cards the other naming steps (possessive, short name, the name
after the comma, alias suffix) point at, under that step's matchedVia, so [[bob's]] offers
Dark Confidant. Dropped as a duplicate when the extractor also
named the card.
- with no naming-step hit, the fuzzy neighbours.
Output: Resolution = Resolved(card, matchedVia) | Ambiguous(candidates) | NotFound
Duplicate references (nickname + full name) are dropped. A span is a
duplicate when it is
- Ambiguous from a non-fuzzy step and shares a candidate with a card
Resolved from another span, or
- NotFound, but its words appear as whole words in a resolved card's
(face) name.
Fuzzy-ambiguous spans are never dropped: trigram neighbours are not
names the user could have meant.
Remaining Ambiguous ⇒ "did you mean…?" buttons and stop. Never guess.
Each loaded card carries its Release (D42): released, or upcoming (first
printed after today) and then covered or not by the loaded CR. Covered
means the CR is dated at most 21 days before the card's first printing
and mentions every type, subtype and rules keyword on it.
│
▼
[3] Classification (same LLM call as [1])
One required primary category plus up to 2 secondary, each with
confidence, from a fixed taxonomy (~25 entries mirroring CR structure).
The schema requires the primary, so an unclassified answer is an API error.
Also assigns Source: CR | Commander | Tournament(MTR/IPG) | OutOfScope.
Tournament/OutOfScope ⇒ reply "I don't cover that" with a pointer.
Flags "nightmare" cards (Humility, Opalescence, Blood Moon, Mycosynth
Lattice, Panglacial Wurm, …) → inject hand-written notes.
│
▼
[4] Retrieval → Context
- CR: category → curated subsection IDs (always) + tsvector BM25 + pgvector
cosine. Union, dedupe, expand to full rule chunk.
Order matters: the synthesis budget (25 chunks / 30 kB) renders
only a prefix, and the sources return several times that. Priority:
1. the primary category's rules sharing a word with the
question (ranked by text relevance, not by id)
2. BM25
3. vector
4. the primary's remaining rules
5. secondary categories (ranked)
`eval recall` gates on what was retrieved (≥ 90%) and on what
that prefix shows (≥ 75%).
- Scryfall rulings for each resolved card (all faces)
- Glossary entries for terms in the oracle text
- Prior calls: vector search filtered by (cards ∩ category), labeled with
rating and CR version, shown as examples AFTER the CR material.
Retired calls are excluded, and so is a call about a card that was not
out when it was answered (D42), for good. The refresh's retirement pass retires a call when a
citation's source no longer contains its quote, or a context card's
Oracle text changed. It restores the call when both hold again.
Renumbered rules carry their calls with them.
- Nightmare notes
- Thread history (last N Q&A)
│
▼
[5] Synthesis (LLM, the model's measured effort: medium on Opus 5.5, high
otherwise; structured output, one tool-use round)
Same per-stage provider choice as [1]. The tool round and the schema
travel in each backend's wire format (strict function calling on
OpenAI-compatible servers when the provider says it has it).
Tool: lookup_rules(ids). The model may request additional CR sections
once before answering, closing the classifier-miss gap.
Output: Verdict { answer, confidence: Low|Medium|High, citations[], category }
The model does not report `source`, `crVersion` or `cards`. Validation
stamps them:
- source from the extraction, as AnswerableSource (by type, only
Cr | Commander reach this step)
- crVersion from the retrieved chunks
- the resolved cards (CardRef: id + name + status) from Context
Every interface then shows "Cards: …" from the verdict alone, and a note
above the answer naming any card not out yet that the loaded CR may not
cover (D42). The material says the same under that card's heading.
Each citation = typed reference + quoted span. Validation:
- first, citations that quote nothing (blank, a stock word, under 4
chars) are dropped (D21). An answer with nothing else is a
MalformedCitation.
(a) the reference exists in Context
(b) the span is a substring of that chunk. Curly and ASCII
punctuation compare equal (judge_core::quote). The chunk's own
text is stored for the span, so a stored quote is exact.
(c) the verdict cites something and the answer is ≥ 40 chars,
else JudgeError.EmptyVerdict
(d) every rule number in the answer text is covered by a rule
citation, else UncitedRules (D20)
The answer text is an Answer newtype: a stray `\n`, `\t` or `\"` the
model wrote as two characters is decoded when the verdict is made (D22).
Failure ⇒ BadCitation / MalformedCitation / EmptyVerdict / UncitedRules →
retry once (the notice says which), then reply with error.
A misused tool round (ToolMisuse: a second lookup_rules call, or one
whose ids cannot be read) produces no verdict and is retried the same
way. The retry may make the one call only if the first could not be
read, since nothing was fetched. A retry after UncitedRules is given the
uncited rules it was not shown (`fetch_uncited`: looked up, or pinned if
retrieval held them and the budget cut them), since a number reached through
a glossary pointer is often not in the material.
Always quotes CURRENT Oracle text (errata note if the printed text differs).
│
▼
[6] Persist call (question, verdict, context ids, crVersion); rating buttons.

The two model calls ([1] and [5]) and the vector search of [4] reach their providers through one seam (docs/PROVIDERS.md).

  • crates/llm holds the provider-neutral request and response, the spend cap and the one-tool-round typestate. ChatRequest has system blocks, turns, tools, an output schema and an effort. ChatResponse has text, tool calls, a stop reason and usage.
    • The cap is a process total plus one adjustment. judge_bot::budget sets it from the spend_days ledger when JUDGE_BUDGET_PERIOD is day or month, which makes the cap a shared budget that survives restarts (D19). It also tells JUDGE_ALERT_WEBHOOK when the cap trips, and when spending reaches a JUDGE_SPEND_WARN_AT threshold (claimed in spend_warnings, so one process sends each). judge-cli stats and the admin dashboard read the same ledger.
  • crates/anthropic and crates/openai are backends of it. Each owns its wire types and its schema-subset transform.
  • The model’s own previous turn is replayed as an opaque blob that only the backend that produced it reads (a thinking signature, a reasoning_content, a tool_calls array). The neutral layer never inspects it.
  • The pipeline’s port, ChatModel, is sealed and implemented only by Metered, so a model that bypasses the cap is unrepresentable. The embedders’ port, WithSpace, is sealed the same way (judge_embed::MeteredEmbedder), on the same meter.
  • Each run also has its own judge_core::Tally (D33). Every port that can spend takes it, and ChatModel::complete takes a Bill naming the run and the Stage. The two metered types add a Charge (stage, provider, the model the response named, tokens, Cost) when they settle with the meter. A response with no usage keeps its reservation on the meter and is charged at it as Cost::WorstCase.
  • Each backend declares Capabilities. What a server cannot enforce (an output schema, strict tools) the adapter moves into the prompt, since decoding and citation validation are client-side either way.

judge_bot::config reads a judge.toml into that: one provider per stage, secrets by environment-variable name, and a price for every model the cap must reserve for. For embeddings it also reads the vector Space (provider kind, model, width). The database records that space in embedding_space, and every vector reader and writer checks it before touching a column. Two models’ vectors are therefore never mixed, and switching is one explicit, transactional ingest reembed. crates/core depends on none of it.

One binary, judgebot (crates/judgebot), runs the long-lived interfaces. What a process does is a set of roles chosen at launch: --discord, --api, --web, --mcp, --admin and --jobs, on the command line or else in JUDGE_ROLES.

  • Role is exhaustive and the set is a NonEmpty, so a process that does nothing is unrepresentable.
  • roles::plan checks every role’s requirements before the pool connects or anything binds, and reports every unmet one at once: --discord needs DISCORD_TOKEN and JUDGE_OPERATOR_DISCORD, the network roles JUDGE_OPERATOR_EMAIL, --web a built page, --mcp an MCP_TOKEN, and --admin an ADMIN_TOKEN of its own (equal to neither MCP token) and a built page. The serving roles’ models are built there too, so a missing or unpriced chat model is reported before the database is touched. What it returns is the types the roles run on, so a role cannot start unchecked. --jobs on its own with the schedule off is refused, since it would have nothing to do.
  • The roles of one process share one composition (judge_bot::serving::Serving): one pool, one Models behind one SpendMeter and one budget ledger, one Vectors, one migration at startup. Each role keeps its own concurrency slots and limits. The HTTP listener is bound before the Discord gateway is contacted, then both run side by side, and the first to stop ends the process with a non-zero status.
  • DISCORD_INTERACTIONS (discord::Delivery, D30) says how Discord delivers interactions. http: the Discord role takes no lease and opens no gateway; it reads the application’s id and public key over REST (retrying until they come), registers the commands, starts the symbol watcher and opens discord::endpoint::Interactions, which the listener serves at /discord/interactions (the network roles’ listener, or with --discord alone one on API_ADDR serving only it and /api/health). A request is verified (Ed25519, 401 otherwise), a PING answered inline, anything else dispatched on a task holding a Work guard, and answered 202 once the handler’s initial response has gone through the REST callback (reply::Ack), at most 2.8 s after arrival. Until the endpoint opens the listener holds its port without listening (at most 30 s), so dials go to another container; a request that finds it closed gets 503, and /api/health fails. Both modes compare the setting with the application’s Interactions Endpoint URL at startup and report a mismatch (ERROR, alert webhook) without stopping. Both feed one dispatch (Data::dispatch, each interaction on a task of its own so a panic is answered like an error).
  • Over the gateway, of the processes running --discord on one database, one holds the gateway (judge_bot::discord::gateway). The Discord role takes a GatewayLease before it connects, or stands by until the holder lets go, with no bound on the wait and reconnecting if its own connection drops; its HTTP roles serve meanwhile. Having taken it, it waits Timing::grace (15 s), unless the holder before it recorded a clean release at shutdown within 5 s (gateway_handover), checks it still holds it, and connects. The holder checks every 5 s (4 s timeout). A failed or unanswered check closes the gateway (discord::shard’s stop: a close frame with 1000, no event after it) and, if it closed within 1 s, stands by again inside the role; otherwise the role ends with an error and the process exits. The grace exceeds the check interval, the check’s timeout and the shutdown limit together (10 s), a relation checked at compile time, so an old and a new holder never answer side by side. A holder in a paused VM that loses its session is the case it cannot cover (docs/DEPLOYMENT.md §8). Every lease session sets server-side TCP keepalives (lease::SESSION_SETTINGS), so a vanished holder’s lock is freed in about 25 s, and turns statement_timeout, idle_session_timeout and transaction_timeout off. A loss soon after the last one pauses before standing by again (30 s doubling to 10 min) to stay inside Discord’s 1000 logins a day per token, and a failed command registration is a warning, never a connection left serving nothing. The role’s Data is one Arc across reconnects, and the symbol watcher a connection starts is aborted with it. hold takes the gateway as a closure handed a Stop, so the hand-off is tested without Discord. The scheduler’s thread is different: beside serving roles its end is logged as an error and the process keeps answering, while a process whose only role is --jobs exits non-zero.
  • SIGTERM or Ctrl-C requests a graceful shutdown (judge_bot::shutdown, D29), one Shutdown per process held by Serving. The HTTP listener closes and answers the requests it accepted (axum’s graceful shutdown). The Discord role closes its gateway and releases the lease the moment the connection is closed, recorded as clean (gateway_handover) when it closed within Timing::shutdown_limit, so the next holder connects without the grace; a standby stops waiting. discord::run takes a Work guard for each interaction as the gateway delivers it, held by the task that answers it, and the role returns once they are all dropped. Everything is bounded by DRAIN_LIMIT (120 s) from the request; compose’s stop_grace_period is that plus EXIT_MARGIN (a test). With both kinds of role, main waits for both rather than the first, then flushes the spend ledger (bounded by FLUSH_LIMIT, 5 s) and exits 0. A gateway slow to close keeps the lease until it has, at most until the limit. A signal before the roles start, or a second one, exits at once.
  • Startup and the schema (D32). migrate::upgrade decides what the pending migrations are. Within a major they only add, so they run first and the process opens (maintenance::Availability::open) beside any previous release still draining. A major upgrade’s (another major, a new major’s X.0.0-* pre-release, or a database from before 2.0) cannot run beside the previous release: the roles start at once, closed (Availability::upgrading), and answer every question with the maintenance reply (Discord: the question restated with a Retry button, render::RetryPrompt, read back from the message as a pick is; /api/judge: 503 maintenance with Retry-After; /mcp: the message as the tool’s error; /api/health: 503 through the Maintenance probe, after the Discord endpoint’s Open so scripts/deploy.sh can read a maintenance: answer as “up”). Every process with a role holds the shared side of SCHEMA_LOCK before it reads the schema, and SchemaHold::keep checks it every 10 s beside the roles (retaken only while the ledger holds nothing unknown and no major holds the exclusive side, else the process exits; a database that does not answer, as while deploy.sh --db replaces it, is tried again every check). The upgrade waits for the exclusive side (lease::SchemaHold::exclusive, in 5 s attempts that give up once another process of its release has migrated), which it gets once every older process has exited, migrates, downgrades to shared, opens, and only then starts the scheduler. A process starting meanwhile queues behind it. A rollback is refused under the migration lock (migrate::rollback_refusal) unless it stays within a major.
  • judgebot ingest <cmd> is the data command line.
  • judgebot backup <run|list|fetch|serve> is the database backup (judge_bot::backup, D26): pg_dump -Fc over the network, gzipped, uploaded to an S3-compatible store (BACKUP_S3_*: endpoint, region, path-style or virtual-hosted) by a SigV4 client of four requests, pruned past BACKUP_KEEP_DAYS after the upload. serve checks the bucket hourly and takes a backup when the newest judgebot-<stamp>.dump.gz is BACKUP_EVERY_DAYS old. It writes what scripts/backup-db.sh writes, so each reads the other’s backups.
  • docker-compose.yml runs the application as one long-running service, judgebot, whose command is JUDGE_ROLES or, unset, --discord --api --web --jobs: roles::COMPOSE_COMMAND, which a test holds the file to. A set API_INTERFACES, the retired variable that chose the old api service’s interfaces, refuses the launch with the JUDGE_ROLES line that replaces it. roles::compose_roles is the same rule in Rust, so judge-config checks only the roles the service would run. It has no container_name and publishes no port, so scripts/deploy.sh can run a second container of it beside the first (blue-green, D29): start the new one, wait for Docker’s healthy, docker stop the old one, which drains. The proxy service (Caddy, proxy/Caddyfile) publishes 127.0.0.1:8787, carries the network alias api (the old service name a tunnel may point at; the runbook’s is proxy:8787) and forwards to judgebot:8787, whichever container listens, on a new connection per request with dial retries for 5 s: a draining container closes its listener, so a refused dial goes on to the other, and no kept-alive connection to it can be closed under a POST. A connection the kernel had queued for it as it closes its listener is reset, which is the only loss that remains. Its :8787 site (the tunnel’s) and its admin site remove X-Judgebot-Client-IP; the host’s 127.0.0.1:8787 is its :8788 site, which passes it on from a reverse proxy on the host (any container on the compose network can dial it too, so nothing there points at it). Both Caddyfiles drain for 70 s (grace_period), under compose’s 80 s stop_grace_period. edge (profile public, proxy/edge.Caddyfile, D41) is the bundled public site: Caddy on ports 80 and 443 serving JUDGE_DOMAIN with a Let’s Encrypt certificate, dialling judgebot:8787 with the same handover settings and setting X-Judgebot-Client-IP to the visitor’s address. cloudflared (profile tunnel) is the alternative route, through proxy:8787. refresh runs judgebot ingest on demand, and backup (profile backup, .env.deploy and no .env) runs judgebot backup serve.

Three interfaces share this pipeline through the same composition root (judge_bot::build_deps):

  • Discord adapter (crates/bot, the --discord role): /judge slash command, rating buttons, “did you mean…?” buttons, thread history.
    • No Discord library (D43): discord::wire is Discord’s JSON as our types (only the fields used; a modal’s components kept as sent), discord::rest the REST calls (a 429 waited out within a bound, nothing else retried, no token in an error), discord::shard the gateway connection (zero intents, READY and INTERACTION_CREATE only; rustls 0.23 with an explicit ring provider), and endpoint::Verifier the Ed25519 check over ed25519-dalek.
    • Every interface hands the store an Origin (judge_core::origin, D31): the interface, the Discord asker (id and username) and whether the call is private. CallStore::persist and record_failure store it; Origin::history_scope says which earlier calls of the thread a question reads.
    • Every user-facing string comes from a per-language catalog (D35): core’s (judge_core::locale: the source offer, the contact, the freshness lines), Discord’s (discord::catalog), the alerts’ (judge_bot::alert::catalog), the API’s (judge_api::catalog), the command lines’ (judge_agent::catalog, judgebot’s catalog), judge-config’s (judge_configure::catalog and its page’s), the web app’s (web/src/messages.ts, web/src/admin/messages.ts) and the docs site’s (site/src/i18n.ts). Each surface picks its value through judge_core::locale::Catalogs::new, which takes every locale’s, the test-only stub locale’s included. A request’s Locale is the asker’s language (the Discord user’s; the browser’s; a terminal’s LC_ALL/LC_MESSAGES/LANG), else JUDGE_LOCALE; an alert is in JUDGE_LOCALE (AlertWebhook::locale). Pick and Retry ids carry the locale their message was written in.
    • Every interface runs a question through judge_bot::run::run (D33): judge, store the call or the failed call, and record the run in judge_runs with its charges. It returns the context the answer was judged from, the run’s Outcome and cost.
    • A rating click (D38) reads the rater’s earlier rating of the call (CallStore::earlier_rating, bounded at 1.5 s from arrival, nothing deferred before it) and judge_core::feedback::next_step decides: the same Correct again is “already recorded”, any other earlier rating gets a private Replace / Keep prompt, else Correct is recorded (deferred, then the judge-role lookup and CallStore::rate) and a fault asks its reason: one modal, written as JSON (discord::feedback::modal: the category menu and both texts in Label components), its values read from the interaction’s components as Discord sent them, however it was delivered (D43). A category menu (rate-cat:) an older build sent over the gateway still opens a modal of the texts in action rows, the category in its id (ids::FormId). The submission defers, looks the role up and stores the rating with its reason (Rating::faulted). Every step’s id carries the call and the score; maintenance answers each with the maintenance reply.
    • /judge-private is Audience::Private: acknowledged ephemerally, no history read, persisted private (with rating buttons), so it is never a prior call and never another question’s history. A call that fails is recorded in failed_calls for any audience through CallStore::record_failure (D27). The audience rides in the pick button’s custom-id through a card pick.
    • A card pick is answered from Discord alone, so any process holding the gateway answers it, after any restart. The prompt message restates the question verbatim above a body with no blank line in it: the lead line, the numbered choices (names cut to 60 characters) and up to two notes (render::PickPrompt, whose parse must re-render the content exactly). Each button’s custom-id (ids::Pick) carries the asker, the audience, the ambiguous span’s byte range in the question, a digest of the prompt’s content (pick::Digest, 8 bytes of SHA-256) and the card’s oracle id.
    • A click does not rely on Discord to check the custom-id against the message. It is refused (as expired) unless the message’s content has the button’s digest, so a button answers only the prompt it was made for, and unless the message’s EPHEMERAL flag agrees with the button’s audience. A message whose flags are missing is answered privately (pick::audience), never recorded. The card must be on the prompt’s list.
    • The prompt’s age is read from the message’s last edit, else its snowflake id, against a ten-minute TTL. pick_claims makes each prompt single-use: one row per message, prompt time and digest, no text, taken after the judge slot so a “busy” leaves the buttons usable, and given back if the acknowledgement fails. Rows older than a day are deleted by the claims that follow. /judge’s question option has a max_length (1300) under which the whole question fits the prompt. Longer content (UTF-16-heavy) gets the choices listed without buttons.
    • The click needs the message’s content. Discord’s interaction object carries “the message they were attached to” for components, ephemeral ones included as far as the docs say, and the message-content restriction exempts messages the app sends. The docs make no explicit promise for ephemeral messages, so a private prompt whose content arrives empty or altered is answered “expired”, never guessed at.
    • A per-user fixed window (discord/cooldown.rs, JUDGE_USER_LIMIT) is charged once a judge slot is held, so a “busy” is free. A card pick is not counted again.
    • /card and /rule are lookups over the resolver, the retriever’s lookup_rules and PgLibrary::rulings. They call no model and touch no meter.
  • HTTP adapter (crates/api + web/): anonymous POST /api/judge behind a per-IP fixed-window rate limit, and a SolidJS web app.
    • The limit’s address is client_ip under API_CLIENT_IP (config::ClientIpSource): the peer, CF-Connecting-IP, or X-Judgebot-Client-IP (ProxyHeader, read only as a single copy from a loopback or private peer), never X-Forwarded-For. Every limiter keys on limit::Bucket: the IPv4 address, or the IPv6 address’s /64.
    • Each interface is a role of its own on one listener (API_ADDR): --api the JSON route, --web the web app and --mcp the MCP transport. An interface nobody named is not mounted.
    • GET /api/health and GET /api/about are served whatever interfaces are off. The container healthcheck needs the first.
    • The public feed (D36): GET /api/feed (keyset pages) and GET /api/calls/{id}, mounted beside the page under --web unless JUDGE_FEED=false, over judge_bot::feed, whose queries are the only public reads of calls (no private call, nothing older than JUDGE_FEED_DAYS). They have their own per-address read limit. The page routes /feed, /calls/<id> and /users/<name> client-side (web/src/router.ts); an unrouted /api/ path is a JSON 404. A user filter (the feed’s and the public dashboard’s) lists the calls of the Discord id behind the name’s newest public call, so a reassigned username does not list its earlier holder’s. A call the feed cannot read in this build is neither shown nor rateable (feed::rateable reads it the way feed::call does). publicly_visible(private, at, window_days), a SQL function the migrations define once and the planner inlines, is the “not private, within the window” predicate of every public read, so the feed’s partial indexes still serve them.
    • The public dashboard (D39): GET /api/stats?from&to&days&interface&user&model, mounted with the feed, over judge_bot::db::stats::public. It counts public calls and runs alone, through publicly_visible: questions per day by interface, outcomes (judge_core::OutcomeGroup), the rating mix (Discord and web apart), the most-asked categories, cards and askers. Cost (CostAccess) and models (ModelAccess) exist only through stats::Disclosure, true only when JUDGE_PUBLIC_COST / JUDGE_PUBLIC_MODELS is; a model filter without models is a 400. A range is 1 to 366 UTC days, in a read-only transaction with a 5 s statement timeout; a range ending before the window is a 400. The reads have their own per-address limit (30 a minute) and two slots per process. The page is /dashboard.
    • /api/about is the source offer (judge_core::source): repository, built commit, licence and copyright. The AGPL requires every remote interface to offer it. The web app’s footer reads it from there. Discord gives the same in /help and /license, and the MCP server in its initialization instructions and an about tool. JUDGE_SOURCE_URL points all of them at a fork.
    • About also carries the data’s freshness (judge_core::Freshness): the CR release loaded and the age of the last successful refresh, and whether the latest one failed. It is read from refresh_runs and rules (ingest::runs::freshness, within 2 s, null and a WARN otherwise; a missing run table leaves the CR version alone). It is cached for a minute, single-flight, by a FreshnessReader: /api/about has one, and every PgLibrary another (/help in the bot, the MCP about tool), so a process with --mcp holds two. /help lists it, the web app’s footer shows it as one line, and /api/health ignores it.
    • The same places name who runs the instance (judge_core::operator). The bot takes a DiscordOperator and the HTTP layer a NetworkOperator. The only way to make either is Operator::for_discord / for_network. So the bot cannot start without JUDGE_OPERATOR_DISCORD, and the network roles cannot start without JUDGE_OPERATOR_EMAIL, whichever of them it opens. A local judge-cli or stdio judge-mcp holds a plain Operator and needs neither.
    • POST /api/calls/{id}/feedback (judge_api::rating, beside /api/judge, D38) rates a public call within the feed window as the tab’s session (Rater::Web, user_id web:<session uuid>), with the address from client_ip and the user agent. The server is authoritative on replacing: an earlier rating without replace: true is a 409 confirm and nothing is written. Its own limiter (rating::LIMIT), a 16 KiB body, unknown fields refused. calls_rated never counts a web rating, because anonymous callers are not accountable identities.
    • /api/judge‘s answer carries call (the stored call’s id). The feed’s ratings count Discord’s and the web’s apart; reasons appear only with JUDGE_PUBLIC_FEEDBACK=true, without the raters’ identity, a dismissed queue item’s left out. Without replace the route writes with rate_if_new (insert only), so a race from one session ends in a 409, never a silent replace.
    • Ambiguity is returned as data and resolved statelessly. The client re-asks with pins: [{span, name}], which the server rewrites to [[Full Name]] with the same pin_card used by the Discord buttons.
    • Follow-up history comes from a client-generated session UUID, stored as thread id web:<uuid>.
  • Admin interface (--admin, judge_api::admin, web/src/admin/, D37): the operator’s view of everything stored, on a listener of its own.
    • ADMIN_ADDR (default 127.0.0.1:8789; the compose container binds 0.0.0.0:8789 and the proxy publishes it on the host’s loopback only, on ADMIN_PORT). Reserved at startup, listening once the instance is open, drained on shutdown. roles::plan refuses an ADMIN_ADDR on API_ADDR’s port.
    • /admin/api/* behind ADMIN_TOKEN (constant-time bearer, no-store): about (JUDGE_LOCALE, the pages’ language when the browser’s has no catalog), calls (every call, private ones included, filtered by interface, asker, model, visibility and text; keyset pages), calls/{id} (origin, ratings with raters and reasons, runs with charges), failures, usage (the operator’s dashboard, D39: stats::usage, every call counted, with the spend ledger by day and month, the budget (the period and cap the ledger-keeping processes record in budget_setting on each sync), the runs by model, interface and stage, cost per answer, latency, the dearest runs and the failed calls; the same range and filters as the public one, the asker by name or id), and the feedback queue (D38): queue (filtered by status, category, interface and rater kind; keyset pages), queue/{id} (the item with its call whole) and its two writes, POST queue/{id}/resolve and …/dismiss with {note, fixed_by}. The queries are judge_bot::admin, judge_bot::feedback and db::stats. The network roles’ listener answers /admin paths 404.
    • /admin and below serve the web app, whose admin pages are a chunk of their own. The page takes the token from the address’s #token= (judge-cli admin-url), keeps it in sessionStorage and sends it only as a header.
    • The same reads are agent operations (admin_calls, admin_call, admin_failures, admin_usage, queue_list, queue_show, queue_resolve, queue_dismiss), each taking a judge_agent::admin::Admin that only Toolbox::from_env (a standalone toolbox) and the MCP_ADMIN_TOKEN handler hand out.
  • Config editor (crates/configure, binary judge-config): a page on 127.0.0.1 for judge.toml and the settings in .env (D23).
    • The judge.toml form is judge_bot::config::file_schema(): JSON Schema generated from the loader’s serde types, with x-endpoints from EndpointKey::on, the table the loader’s misplaced-key check reads.
    • env::VARS lists every variable .env.example carries as a secret or a setting. A value reaches the page only through a Setting. A secret, a variable the list does not know, or a hidden setting is reported as set or not, and is changed write-only (DotEnv::replace): the page sends a new value and every reply is scrubbed of it. Help text is .env.example’s comments.
    • Each draft goes through check::run: the judge.toml loader, discord::Config::from_vars, ApiConfig::from_vars and check, the migration flag. Errors carry ConfigError::location(), the key or variable to fix.
    • Writes go through toml_edit and a line-preserving .env writer, so only changed lines move. They are refused when a file changed since it was read.
  • Agent adapter (crates/agent): the judge as a tool surface for other agents.
    • Over MCP, judge-mcp serves a local client on stdio. For a remote one, judgebot --mcp mounts the same handler at /mcp behind MCP_TOKEN. The flag without a token is refused at startup. The token without the flag serves nothing and warns.
    • The operator’s reads are operations too (D37): admin_calls, admin_call, admin_failures, admin_usage (judge-cli calls, call, failures, stats), and the feedback queue’s (D38): queue_list, queue_show, queue_resolve, queue_dismiss (judge-cli queue, queue-show, queue-resolve, queue-dismiss). Each takes an admin::Admin, which judge_agent constructs only beside a standalone toolbox (Toolbox::from_env returns Local) and in mcp::admin_http_service; an instance toolbox (Client::instance) gets none. What a connection may call is its handler’s type: JudgeMcp<Public> (what MCP_TOKEN reaches) has no admin tools, JudgeMcp<Privileged> (stdio judge-mcp, and /mcp with MCP_ADMIN_TOKEN) adds them, and each admin call is logged at INFO with its tool’s name. judge_api::mcp::router takes the public service by that type; the bearer scheme is matched case-insensitively.
    • judge-cli has one subcommand per operation with JSON out, for a shell agent (the repo’s .claude/skills/judge skill).
    • It offers the pipeline two ways:
      • The judge tool runs it as above, with the built-in model calls. It is spend-capped, shares the web route’s concurrency semaphore, and is offered only when a model is configured (ANTHROPIC_API_KEY or a judge.toml).
      • A session runs it in pull mode, where the calling agent is the model.
    • judge_bot::session is that state machine.
      1. begin returns the extraction prompt (steps 1 + 3 as text, plus the JSON Schema).
      2. The agent’s Extraction JSON drives steps 2 + 4 and yields the synthesis prompt: the same system prompt, with Harness-specific wording for the one lookup_rules round and the output format, plus the rendered material.
      3. The agent’s Verdict JSON goes through the same Verdict::validate against the session’s own Context, with the same one retry and the same rejection notice.
    • On the API path, typestates carry the invariants (one tool round, one retry, only a validated verdict is persisted). Here they are a Stage enum, because the state lives in Postgres between calls (agent_sessions, one jsonb document, optimistic version). The session id is a server-minted handle passed as a tool argument, which is what the 2026-07-28 MCP revision adopted in place of protocol sessions. The HTTP transport is served statelessly for every protocol version.
    • Sessions are unauthenticated at the tool level. Their thread ids are therefore a type (AgentThread, always agent:<uuid>) that cannot name a Discord or web thread, and their inputs are bounded (question, spans, concepts, lookup ids, answer length). A persisted call is keyed by session (calls.session_id, unique), so persisting twice cannot file two calls. It is excluded from the prior-call query: nothing can rate it, and it is history for its own thread only.
    • Over HTTP, judge runs are also capped per window (MCP_JUDGE_LIMIT) so a leaked token cannot take the public web app’s slots and spend cap with it.
    • Read-only lookups (resolve a card, rules by id or search, rulings, notes, glossary) are the rest of the surface (PgLibrary).

The Discord and web interfaces draw Magic’s card symbols ({W}, {2/U}, {T}) as pictures, from one set of names:

  • Discord substitutes application emoji (<:mana_w:…>). The bot owns them, not a server, so they work in every guild and cost no emoji slots.
    • ingest emoji uploads them: Scryfall’s SVG → a 128 px PNG via resvg, scaled to fit and centred (nine of the 84 symbols are not square).
    • The emoji name is defined once, in judge_core::symbol, with no I/O. It is in core because the uploader and the renderer are separate programs that must agree on it exactly. A test pins the mapping as total and injective over everything Scryfall publishes, so no symbol can silently overwrite another’s emoji.
    • A tag is ~28 characters where {W} is three, and Discord counts the tag. mana::Rendered keeps text as segments and lets only plain text be cut, so a half-written tag is unrepresentable rather than tested against.
    • An application with no emoji uploaded gets an empty table and the literal {W}, unchanged.
    • The table sits behind a swappable mana::SharedSymbols; a render takes a snapshot. discord::symbols::watch, a task spawned on Ready so connecting never waits on it, reads the run record’s mark, lists the emoji, then every ten minutes reads refresh_runs for a run finished since the mark whose emoji step may have uploaded (runs::emoji_since: anything but skipped or uploaded: 0). It lists them again after such a run, after a failed listing, while the table is empty, and hourly regardless, which catches uploads and deletions the record does not show. So a refresh by any process (a --jobs role, a cron’d or manual judgebot ingest) or an upload by hand reaches a running bot without a restart.
  • Web renders Scryfall’s SVGs inline from their CDN (web/src/symbols.ts is the generated table, Symbols.tsx the component).
    • A symbol the table does not know, or an image that fails to load, falls back to the literal text.
    • split/lookup mirror the Rust scanner, so both surfaces accept the same spellings ({W/U}, {w/u}, {U/W}, {WU}) and leave the same text alone.
    • Seven symbols ({E} {P} {PW} {CHAOS} {TK} {L} {D}) are flat black with no disc, invisible on the dark palette. They carry a flat flag and are inverted in dark mode. The coloured ones must not be.
    • If a Content-Security-Policy is ever added to the HTTP adapter, img-src must allow https://svgs.scryfall.io.
Source Refresh Storage
Scryfall bulk oracle-cards.json daily (the scheduled refresh judgebot --jobs runs, judge_bot::jobs, every JUDGE_REFRESH_HOURS — DEPLOYMENT.md §7) cards (oracle_id, name, layout, type_line, first_printed — the earliest printing’s date from default-cards, D42 —, …) + card_faces (oracle_id, face_idx, name, oracle_text, mana_cost, …)
Scryfall bulk default-cards.json (names only) daily printed_names (printed_name, oracle_id) — old names, errata’d names
Scryfall bulk rulings.json daily (bulk-loaded, keyed by oracle_id) rulings (oracle_id, key, published_at, text) — key = content hash (judge_core::ruling_key), so a reindexed ruling keeps its identity
Comprehensive Rules txt on CR release — detected daily from the .txt link on Wizards’ rules page vs max(cr_version) rules (id, parent_id, subsection, heading, body, examples, embedding, cr_version)
CR Glossary same glossary (term, text, embedding)
Scryfall /symbology (84 card symbols) daily (idempotent, uploads only missing symbols) not stored: uploaded as Discord application emoji (ingest emoji) and hard-coded for the web app (web/src/symbols.ts)
Nicknames hand-curated YAML, compiled in; the refresh reloads a built-in copy an upgrade changed card_aliases (alias, oracle_id)
Nightmare notes hand-written markdown, likewise card_notes (oracle_id, note)
Curated list sources written by each alias or note load, in its transaction curated_lists (list, source builtin/file, digest, loaded_at): the refresh’s lists step reloads only a builtin list whose digest is not the binary’s (judge_bot::ingest::lists)
Categories → subsections YAML (single source of truth; the enum is generated from it) categories
Calls continuous; retired_at/retired_reason recomputed on each refresh from citation validity; deleted with their ratings past JUDGE_CALL_RETENTION_DAYS by the refresh’s prune step (D34) calls (id, thread_id, interface, asker_id, asker_name, private, locale, question, answer, category, citations jsonb, source, cr_version, retired_at, retired_reason, embedding, unreleased_cards — the cards its reply noted as unreleased, D42; a private call has no embedding)
Failed calls on each operator failure; pruned to 30 days / 500 rows on write failed_calls (thread_id, interface, asker_id, asker_name, private, question, error, first_rejection, attempts)
Judge runs one row per built-in pipeline run, any outcome (D33) judge_runs (started_at, finished_at, interface, private, outcome, retried, tool_round, call_id, failed_call_id) + judge_run_charges (run_id, seq, stage, provider, model, tokens, micro_usd, cost_estimated)
Ratings continuous; a web rating’s address and user agent cleared past FEEDBACK_IP_DAYS by every rating write and the refresh’s sweep step (D38) ratings (call_id, user_id, score, is_judge, ts, rater_kind discord/web, category, what_wrong, right_answer, web_name, ip, user_agent)
Feedback queue one item per rating given a reason; reopened by a replaced reason feedback_queue (id, call_id + user_id → ratings, status open/resolved/dismissed, note, fixed_by, opened_at, resolved_at)
Refresh runs one row per refresh, scheduled or judgebot ingest refresh refresh_runs (started_at, finished_at, trigger, process, cr_before, cr_after, steps jsonb, ok)

The loaders in the table (cards, rulings, the CR, symbols, nicknames, notes) and the embedding step are judge_bot::ingest (crates/bot/src/ingest/), beside the other Postgres adapters. judgebot ingest is the command line over them.

The database has three advisory lock keys, listed together in judge_bot::lease with a compile-time check that they differ:

Key Scope Held by
REFRESH_LOCK session a data-writing run (RefreshLease)
GATEWAY_LOCK session the process connected to the Discord gateway (GatewayLease)
CALLS_REWRITE_LOCK transaction, shared or exclusive; session for a migration a persist that writes a vector and each embed batch (shared); the CR load, the retirement pass, a space switch (exclusive); a migration, for its whole run

The two session locks are a Lease<K>, typed by what they guard (K is Refresh or Gateway, a sealed LeaseKey carrying the key, the application_name label and, in its own impl, the wait), so one lease cannot stand in for the other.

Every loader that writes the database takes a RefreshLease, a session-level advisory lock (REFRESH_LOCK) held on a connection of its own, so two runs never overlap, in one process or several, and a step without it does not compile. Steps take it as &mut RefreshLease, so two cannot run at once under one lease either. Waiting for it is bounded (an hour), and a run checks before each step that its session still holds it. Postgres drops the lock with the session, so a crashed run leaves nothing to clean up. It is separate from CALLS_REWRITE_LOCK, the short transaction-scoped lock the CR load, the retirement pass and vector writes take inside a run (and a migration for its whole run). A run takes the lease first and the calls lock inside it, never the other way round.

Every ingest::refresh, whatever started it, checks before each step that it still holds the lease and that the migration ledger matches its binary (migrate::skew), and stops at ingest::RUN_TIMEOUT (3 h), abandoning the step in progress. A schema change skips the remaining steps and the run is RunOutcome::Stopped, stored with ok null: neither a success nor a failure, and not an attempt that backs the schedule off. A single-step judgebot ingest command makes the same check once (ingest::ensure_writable). runs::history counts an unfinished row older than runs::ABANDONED_AFTER as a failed run, so a process that dies mid-run still lengthens the failure streak.

A scheduled run also stops at its process’s shutdown (D40). ingest::refresh takes a Caller: Caller::Schedule(&Shutdown) from the scheduler, Caller::Manual from judgebot ingest refresh. Once the shutdown is requested the step in progress is cut off at its next await point (Skip::Interrupted), the rest are skipped (Skip::Shutdown), and the run is Stopped, whatever its earlier steps did. Every step is safe to cut off: each writes in transactions or idempotent statements, and downloads are renamed into place whole (the table in D40). emoji_since counts an interrupted emoji step as one that may have uploaded.

The schedule is judge_bot::jobs, which a process with --jobs starts after migrating. It runs on an OS thread of its own, with a current-thread runtime and a three-connection pool, so a run’s file I/O, CR parse and queries never take a worker or a pooled connection from the request path. The database is shared, though: while the CR load or the retirement pass holds the exclusive side of CALLS_REWRITE_LOCK, a PgCallStore::persist that writes a vector waits on the shared side, and both interfaces persist before they reply. A cron run has the same effect.

Every ten minutes or so the scheduler reads the migration ledger (a schema ahead of or behind the binary pauses it), then refresh_runs; a database with no rules waits for init, which records itself as a manual run, so the first scheduled refresh comes one interval after it. The pure jobs::due decides on the database’s clock: due when the last success is older than JUDGE_REFRESH_HOURS and the last attempt older than jobs::backoff (1 h, doubling per failure in a row, capped at the interval). When due it tries the lease without waiting, re-reads the record under it and runs ingest::refresh with Trigger::Schedule. That trigger also caps embed at embed::UNATTENDED_CEILING rows (Skip::EmbedCeiling), so an unattended run cannot pay for a mass re-embed. The webhook (judge_bot::alert, shared with the spend cap) is sent the first failure of a streak, the recovery, a new ceiling skip and a panicking check. D24 records why the schedule is in the process and not in cron.

jobs::start takes the process’s Shutdown. On the request the scheduler starts no check, every wait outside a step is dropped (ingest::unless_stopped), the run in progress stops as above, its row is closed and the lease released (those writes share one deadline, ingest::STOP_WRITES after the request: ingest::until_stopped), a run that finished before the request still posts its alerts, and the thread ends (Ended::Shutdown), shutting its runtime down so every connection it opened closes. Its connections carry client_connection_check_interval (jobs::CLIENT_CHECK_INTERVAL, 2 s), so a statement the cut step left running on the server ends within seconds of that, and with it the transaction and any CALLS_REWRITE_LOCK it held. main waits for the thread after the roles drain, at most until the drain’s limit (Scheduler::stopped), then writes the spend ledger and exits. jobs::STOP_LIMIT (everything after the request) is a const under DRAIN_LIMIT.

The record is also what users and the operator see of it. About carries the CR release loaded and the age of the last successful run (see the HTTP adapter above), and judge-cli stats lists the last five runs (runs::recent) with their outcome and failed steps.

Database: Postgres 16 + pgvector + pg_trgm. Scale: ~30k cards, ~2k rule chunks, <10k calls.

Embeddings: Voyage AI by default (voyage-3.5 or voyage-4 family; voyage-3 is superseded). It was chosen over local models for the zero-config setup, because local models are not worth it on WSL2. Embedder is an interface, so this can change: judge_embed also has an OpenAI-compatible /v1/embeddings adapter for local or hosted models, chosen by [models.embed] in judge.toml.

Every embedder carries its Space (provider kind, model, width). The one-row embedding_space table records the space the stored vectors belong to, and ingest embed refuses to write into another. On a mismatch the adapters turn vector search off (error log, never mixed). The space is re-checked on every use, and anything that writes a vector holds it under a shared advisory lock. ingest reembed --yes switches the database in one transaction after probing the new embedder (docs/PROVIDERS.md §4.3).

Rating aggregation: Bayesian-smoothed mean (prior 2.0, weight 3) over Discord ratings only (calls_rated reads rater_kind = 'discord'). A rating with is_judge = true (operator-assigned role) dominates crowd votes. It is used for labeling and ordering among prior calls, and for exclusion (< 1.5, with ≥ 5 votes unless a judge rated it). It never ranks prior calls above CR chunks. A rating’s reason is read by the operator’s queue alone.

The model is language-neutral.

CardId = Scryfall oracle_id
RuleId = "702.19" | "702.19b"
Face { name, oracleText, manaCost, typeLine }
Card { id, name, layout, faces: NonEmpty[Face] }
Category closed set, ~25 values, sourced from categories.yaml
Source CR | Commander | Tournament | OutOfScope
Confidence Low | Medium | High
Resolution Resolved(card, matchedVia) | Ambiguous(query, candidates) | NotFound(query)
Quote non-blank text; compared to its source with typographic punctuation folded
Citation Rule(id, quote) | ScryfallRuling(card, rulingKey, quote)
| OracleText(card, quote) | PriorCall(id, quote)
Context { cards, rules, rulings, glossary, prior, notes, history }
Verdict<S> { answer, confidence, citations, category, source, crVersion, cards }
S = Unvalidated | Validated; only Verdict<Validated> can be stored or shown
Rejection BadCitation(citation) | Malformed(what) | Empty(why) | Uncited(rule ids)
| Tool(SecondRound(ids) | Unreadable(what)) | Oversized { chars }
RejectedAttempt { answer, rejection } — quoted back to the retry as a blockquote
JudgeError AmbiguousCards(...) | CardsNotFound(...) | OutOfScope(source)
| BadCitation | MalformedCitation | EmptyVerdict | UncitedRules
| ToolMisuse | LlmRefused | Upstream(err)
Ports: Extractor, Resolver, Retriever, Synthesizer, Embedder, CallStore
judge : Question -> IO[Either[JudgeError, Verdict]]

The build was eval-first: retrieval was measured before any synthesis existed.

  1. A gold set of adversarially verified questions with expected rule ids (eval/gold.yaml, 24 questions today, one of them a human correction and two about unreleased cards).
  2. Ingest.
  3. Extraction and resolution, tested on the gold set’s card mentions.
  4. Retrieval, behind a gate of ≥ 90% of gold rule ids present in the Context.
  5. Synthesis with citation validation, scored against the gold answers.
  6. The Discord adapter with rating buttons.
  7. The prior-call query, which needs rated data to exist.

judge-eval recall still runs that gate for free on every retrieval change.

  • Tournament policy (MTR/IPG). The bot declines those questions.
  • Accounts on the web app. Its visitors are anonymous, so a rating from it would never count toward retrieval (D10).
  • Retraining of any kind.
  • Automatic detection of “nightmare” cards (the notes are curated by hand).
  • Multi-server tenancy. The bot is meant to be run by each community for itself (docs/DECISIONS.md D16), so one process has one spend cap, one judge role and one token.