Development setup and gates
This page covers the development environment, the gates a change has to pass, and the
design rules that are not obvious from the code. docs/EXPLAINER.md explains how the bot
works end to end. docs/ARCHITECTURE.md is the design reference.
Development setup
Section titled “Development setup”You need Docker (with the compose plugin), the Rust release
rust-toolchain.toml pins (rustup installs it on first use) and the Node major the
Dockerfile’s node: image names, for the web app.
cp .env.example .env # DATABASE_URL already points at the compose database # (later edits: cargo run -p judge-configure, the config editor)docker compose up -d db # pgvector Postgres on localhost:5432set -a; source .env; set +a # optional: the binaries read .env themselves; this puts it in your shell toocargo build --workspacecargo test --workspace # the #[sqlx::test] suites create throwaway databases off DATABASE_URLThe database publishes on loopback port 5432. If something on your machine already
has it, set DB_PORT in .env and change the port in DATABASE_URL to match. The
compose file reads the same variable.
Nothing in the test suite calls a paid API. HTTP backends are tested against wiremock.
Develop against it too, and keep ANTHROPIC_API_KEY blank unless you mean to spend.
Card and rules data is needed to run the pipeline, not to build or test.
cargo run --release -p judgebot -- ingest init loads all of it from source. The README’s
“Running it” does the same in a container. Every docker compose run --rm refresh <command> in the docs is cargo run --release -p judgebot -- ingest <command> here.
judgebot is the one long-running binary, and what it runs is chosen by its flags:
cargo run -p judgebot -- --api --web serves the web app on localhost:8787, --discord adds
the bot, and --jobs the scheduled data refresh. Leave --jobs off, or set
JUDGE_REFRESH_HOURS=0, against a development database you do not want refreshed.
cargo run -p judgebot -- --help lists the roles and what each requires.
The gates
Section titled “The gates”CI (.github/workflows/ci.yml) runs these on every pull request, and the image is
published only from a tree that passes them. scripts/check.sh runs the same gates
locally, and git hooks run it for you:
scripts/dev-setup.sh # once per clone: the pinned linters, npm ci in web/ and site/, the hooksBoth hooks check exactly what is being committed or pushed, not your working tree. An
unstaged fix, a file you forgot to add or another branch’s state cannot make them pass.
They check the tree out into a scratch worktree under .git/:
- It shares
target/and the linters with your clone. - It has its own
node_modules, reinstalled when a lockfile changes. - It gets
.env.exampleas its.env, as CI does.
The .sqlx check migrates a throwaway judgebot_check database on your server, never
your development one.
- pre-commit checks the staged tree. It runs formatting and clippy when Rust or
compiled-in data is staged, Biome and the build for
web/, and Biome,astro check, the build and a link check for the docs site. It always runs the repository lints:cargo deny(against the advisory database already fetched, so it works offline),cargo machete,taplo,typos,shellcheck,actionlint,hadolint, a check that the icon’s copies match, a check that every Rust and Node version agrees with theDockerfile’s, and the compose file. It takes seconds unless clippy has a lot to recompile. - pre-push checks the tip of each ref being pushed against the remote’s commit. It
runs the repository lints, and, when Rust, compiled-in data,
eval/or the compose file changed,cargo test, the migrations and the.sqlxfreshness check (fmt, clippy, web and site already ran at commit). These need the database up (docker compose up -d db) andsqlx-cli. A change toscripts/check.sh,scripts/tools.shor a workflow, a new branch, or a remote commit your clone has not fetched gets every group. CI runs the same script. Commits that skipped pre-commit (--no-verify, or replayed by a rebase or cherry-pick) get their fmt, clippy, web and site checks only in CI, so runscripts/check.shbefore pushing after a rebase.
Other ways to run the gates and linters:
scripts/check.sh rust testruns chosen groups on the working tree (any ofrust sqlx test web site lint).scripts/check.sh --at <rev>runs them on a commit.sqlxandtestwait for any othersqlxortestrun on the same Postgres server.#[sqlx::test]names its throwaway databases after each test’s path, so twocargo testruns on one server at once break each other: while a hook may be testing, run the suite asscripts/check.sh test.git commit --no-verify/git push --no-verifyskip a hook once.npm --prefix web run fix(orsite) applies Biome’s formatting and safe fixes..tools/bin/taplo fmtformats the TOML.scripts/clean-target.sh [--days N] [--dry-run]reclaimstarget/: it deletestarget/debug/incrementaland, withcargo install cargo-sweep, artifacts older than N days (default 14). Run it weekly and after a toolchain bump, a profile change or a large dependency update.scripts/check.shprints a note whentarget/passes 100 GB.
The linters are prebuilt binaries at the versions in scripts/tools.sh, which CI installs
too. They live in .tools/ (gitignored). Bumping a version there is the whole upgrade.
Workspace lints deny unwrap, expect, indexing and panic! in every crate, and warn
on missing docs and the pedantic group. Reach for a type or a Result instead of a
suppression. When one is right, keep it on the one item as
#[expect(lint, reason = "…")]. A bare #[allow] is a build error, and an expect
that stops matching anything fails the build too, so a stale suppression cannot linger.
deny.toml is the dependency policy. It covers RustSec advisories, the licences an
AGPL image may carry, and crates.io as the only source. A dependency with a new licence
fails CI until the licence is added there. An accepted advisory is listed there with its
reason.
SQL changes. Queries are checked at compile time against the schema. After editing
any sqlx::query! or migration, regenerate the committed offline data and include it
in the same commit. CI fails when it is stale.
cargo sqlx prepare --workspace -- --all-targets # needs the database up and migratedSQLX_OFFLINE=true cargo build --workspace # must pass without a databaseInstall the tool with cargo install sqlx-cli --no-default-features --features postgres,rustls if you do not have it.
Within a major version, migrations only add. A deploy starts the new container,
which migrates, while the old one is still answering (scripts/deploy.sh,
docs/DECISIONS.md D29), and a judge-cli or judge-mcp of an older minor keeps
working against the new schema (judge_core::release). So every earlier release of the
same major must still work against it:
- Fine in a minor or patch release: a new table; a new column that is nullable or has a default; a new index; a new constraint that earlier releases’ writes already satisfy. A rename or a type change is a new column the release writes beside the old one (or reads with a fallback), and the old one stays.
- Only in a major release: dropping or renaming a column or table, changing a
column’s type, making a column
NOT NULLwithout a default, and rewriting rows that earlier releases read or write (moving stored citations to a new key, say). This is also where the old shapes the minors left behind are dropped. Say what changed under the release’s### UpgradinginCHANGELOG.md.
A major release’s deploy stops the old container before the new one migrates, so it has a short downtime. Its migrations must still finish quickly, because they run inside that window: a slow backfill belongs in a background job.
A released migration is never edited.
Prompt and schema changes. Two tests pin the bytes the bot sends. If you edit a
prompt template under crates/bot/src/prompts/ or a struct that feeds the model’s output
schema, both fail until you update them:
crates/bot/tests/anthropic_golden.rscompares four request bodies to fixtures. Run it withUPDATE_GOLDEN=1to re-capture them, then read the diff of the fixtures as part of your review.harness_testsincrates/bot/src/synth.rspins the SHA-256 of the tuned synthesis system prompt. Update the digest in the test when the change is intended.
Taxonomy changes. data/categories.yaml generates the Category enum at build
time (crates/core/build.rs), so editing it is a recompile, and every match over the
enum has to be updated. data/aliases.yaml and data/notes.yaml are compiled in as
data and loaded into the database by judgebot ingest. A deployment that loaded the
built-in copy picks up an edit on its first refresh after the upgrade. A change to their
entries, not only comments, also copies the previous file into data/legacy/ and appends
it to LEGACY in crates/bot/src/ingest/aliases.rs or notes.rs, so a database loaded
before the source record existed is still recognised as built-in.
Gold set changes. eval/gold.yaml quotes every rule id (unquoted 702.10 is a
float and is rejected). cargo run -p judge-eval -- recall is the free retrieval gate.
rescore re-scores stored runs after a gold edit without spending anything.
Model and effort comparisons. A comparison is a plan file in eval/sweeps/ run
with judge-eval sweep, not a series of hand-run answer commands: the plan names the
models, efforts and repeats, the sweep grades every run with one model and averages
them in summary.md. --dry-run first.
Design rules
Section titled “Design rules”The full statement is docs/DECISIONS.md D1. These are the ones a change is most
likely to break.
- Make the bad state unrepresentable. Invariants live in types: closed enums,
validated newtypes,
NonEmpty, and typestates such asVerdict<Unvalidated>→Verdict<Validated>and the synthesis loop’s single tool round. Prefer that over a runtime check, and a runtime check over a test. crates/corehas no I/O. No reqwest, sqlx, tokio-net or provider SDK may be added to it. That dependency fence stands in for effect tracking.- Every model call is metered. The only way to send a request is through
judge_llm::Metered, which reserves the worst-case cost againstJUDGE_MAX_USDfirst, or, for an embedding,judge_embed::MeteredEmbedderon the same meter. Do not add a second path to a provider. - Citations are validated. A quote must be a substring of its source in the retrieved
context. Only a
Verdict<Validated>can be persisted or shown. - The resolver never guesses. Ambiguity in a card name becomes a “did you mean” prompt, not a best effort.
- Secrets are
ApiKey. Anything that authenticates (provider keys, the Discord token, the MCP bearer token) is wrapped so aDebugof any config prints<redacted>. A user’s question text does not go to logs abovedebug. The card names extracted from it do.
Translations
Section titled “Translations”Every user-facing string lives in a catalog, one value per language
(docs/DECISIONS.md D35). A new message is a field on its surface’s Messages, with a
value in every catalog of that surface, the stub’s (STUB: its own name in ⟦…⟧ with
its values) included, and a line in that surface’s every_locale_renders_every_message
test. Diagnostics (logs, the command lines’ errors, health checks), JSON output, the
model prompts and the agent tools’ text stay English; the replies an interface sends a
person, errors included, are catalog text.
Adding a language:
- The locale. In
crates/core/src/locale.rs, add theLocalevariant, itsid()(the language subtag, e.g.es), its place inALL, and its parameter and arm inCatalogs::newandCatalogs::get. Every surface’sCatalogs::newcall then stops compiling until it names the new catalog. - The Rust catalogs. Write one
Messagesvalue per surface; the compiler lists every missing message:judge_core::locale(the source offer, the contact, the freshness lines);judge_bot::discord::catalog, and the language’s Discord locale tags inTAGS(its command descriptions register under them);judge_bot::alert::catalog(the alert webhook);judge_api::catalog(the HTTP API’s messages and citation labels);judge_agent::catalogandcrates/judgebot/src/catalog.rs(the usages);judge_configure::catalog, and the schema and.envhelp overrides beside it.
- The pages. Add the language to
LOCALESand a catalog toCATALOGSinweb/src/messages.tsandweb/src/admin/messages.ts(tsclists what is missing), an object to judge-config’s page catalog,crates/configure/ui/messages.js, in the format its header describes (cargo test -p judge-configureholds its keys to English’s), and to the docs site: the locale insite/src/locales.mjs, its strings insite/src/i18n.ts, sidebar labeltranslationsinsite/astro.config.mjs, and its pages undersite/src/content/docs/<locale>/(authored, or manifest entries withlocale). - Limits.
scripts/check.shruns the tests that hold every locale to Discord’s limits (100-character command descriptions, the rating form’s 45 and 100) and theconstassertion that/judge’s question still fits a “did you mean?” prompt (render::PICK_QUESTION_LIMIT, the least over every catalog): a wordier lead shortens the question a prompt can hold, and the build says so.
Debugging a wrong answer
Section titled “Debugging a wrong answer”The cheapest way to reproduce a bad ruling is the agent session. It runs the bot’s
extraction, retrieval and citation validation with you as the model, and it spends
nothing. Build judge-cli (cargo build --release -p judge-agent), bring up the
database, and step through judge-cli begin | extract | rules | verdict as
.claude/skills/judge/SKILL.md describes. judge-cli card, search, get-rules and
glossary query the database directly. Use judge-cli judge or a live bot only when
the deployed surface is in question, because those calls cost money.
Pull requests
Section titled “Pull requests”- Keep a pull request to one change, with the reason in the description. Tests and
the
.sqlxor golden fixtures it needs go in the same commit as the code. - The changelog is
CHANGELOG.md, Keep-a-Changelog style. Add a line under Unreleased for anything an operator or user would notice. - Card names, rules text and rulings in tests should be short excerpts. Do not commit
data dumps,
.envfiles,judge.tomlor anything undereval/runs/.
Releases
Section titled “Releases”A release is a GitHub release whose tag is vX.Y.Z on a commit of main.
Between releases, main carries the next version with a -dev suffix (2.0.0-dev),
set in the same places as step 1 below when work toward that release starts. Each
migration records the release and commit that ran it in instance_release, and
judge-cli, judge-mcp and judgebot itself compare their own release with it
(judge_core::release):
- Within a major (
2.1.0-devincluded), migrations only add, so builds work together in either direction. An older tool prints a note, and an olderjudgebot(a rollback) starts. - A new major’s pre-release (
3.0.0-dev,3.0.0-rc.1) is where its breaking migrations are made, so it works only with the build that migrated the database. Whilemainis on one, switching a checkout back to an earlier commit means a fresh development database (or a backup of one):judgebotrefuses the one a later build migrated. - Across a major, or against a database missing one of their migrations, the
tools stop and
judgebotrefuses to start, each saying what to update.
- Move the Unreleased changelog section under a
## [X.Y.Z]heading, and set that version in[workspace.package]in the rootCargo.toml(thencargo update --workspaceforCargo.lock), inweb/andsite/‘spackage.jsonandpackage-lock.json, and in the image-tag examples indocs/DEPLOYMENT.mdand the configuration page. The binaries report the crate version (the MCP server, the ingest steps’ User-Agent).scripts/release-version.shchecks the copies:check.sh lintthat they agree, andpublish-image.ymlthat they are the tag’s and the changelog has its heading. A release that fails it publishes no image: delete the release and its tag, fix the version, and create both again. - Wait for the push’s Publish image run to finish. Otherwise the release and that run build in parallel.
- Create the release (
gh release create vX.Y.Z --generate-notes, or the web form).
Publishing it runs publish-image.yml. That tags the image already built for the commit
as X.Y.Z, X.Y and, from 1.0 on, X, without rebuilding. So the release is the same
image that has been running as latest, down to the platform digests. A commit with no
image of its own (a docs-only commit, or a build a later push cancelled) is built first,
through CI. Pre-release tags such as v1.0.0-rc.1 get only their exact version tag, and
need that exact version in the tree (step 1) first, because the binaries report it and a
new major’s pre-release works only with its own build.
latest keeps following main.
Claude Code
Section titled “Claude Code”The repository carries a .mcp.json and a judge skill under .claude/skills/. With
Claude Code, they let Claude act as the model in the judge pipeline through judge-mcp.
Other tools ignore them. CLAUDE.md is Claude’s working notes for this
codebase. It is kept in sync with the docs, but the docs are the source of truth.
MTG Judgebot is unofficial Fan Content permitted under the Fan Content Policy. Not approved or endorsed by Wizards of the Coast. Portions of the materials used are property of Wizards of the Coast. ©Wizards of the Coast LLC.
The Comprehensive Rules come from Wizards of the Coast. Card data, rulings and card symbols come from Scryfall, which is not affiliated with this project. Rule links go to the Yawgatog mirror. License and attribution.