Skip to content

Development setup and gates

This page covers the development environment, the gates a change has to pass, and the design rules that are not obvious from the code. docs/EXPLAINER.md explains how the bot works end to end. docs/ARCHITECTURE.md is the design reference.

You need Docker (with the compose plugin), the Rust release rust-toolchain.toml pins (rustup installs it on first use) and the Node major the Dockerfile’s node: image names, for the web app.

Terminal window
cp .env.example .env # DATABASE_URL already points at the compose database
# (later edits: cargo run -p judge-configure, the config editor)
docker compose up -d db # pgvector Postgres on localhost:5432
set -a; source .env; set +a # optional: the binaries read .env themselves; this puts it in your shell too
cargo build --workspace
cargo test --workspace # the #[sqlx::test] suites create throwaway databases off DATABASE_URL

The database publishes on loopback port 5432. If something on your machine already has it, set DB_PORT in .env and change the port in DATABASE_URL to match. The compose file reads the same variable.

Nothing in the test suite calls a paid API. HTTP backends are tested against wiremock. Develop against it too, and keep ANTHROPIC_API_KEY blank unless you mean to spend.

Card and rules data is needed to run the pipeline, not to build or test. cargo run --release -p judgebot -- ingest init loads all of it from source. The README’s “Running it” does the same in a container. Every docker compose run --rm refresh <command> in the docs is cargo run --release -p judgebot -- ingest <command> here.

judgebot is the one long-running binary, and what it runs is chosen by its flags: cargo run -p judgebot -- --api --web serves the web app on localhost:8787, --discord adds the bot, and --jobs the scheduled data refresh. Leave --jobs off, or set JUDGE_REFRESH_HOURS=0, against a development database you do not want refreshed. cargo run -p judgebot -- --help lists the roles and what each requires.

CI (.github/workflows/ci.yml) runs these on every pull request, and the image is published only from a tree that passes them. scripts/check.sh runs the same gates locally, and git hooks run it for you:

Terminal window
scripts/dev-setup.sh # once per clone: the pinned linters, npm ci in web/ and site/, the hooks

Both hooks check exactly what is being committed or pushed, not your working tree. An unstaged fix, a file you forgot to add or another branch’s state cannot make them pass. They check the tree out into a scratch worktree under .git/:

  • It shares target/ and the linters with your clone.
  • It has its own node_modules, reinstalled when a lockfile changes.
  • It gets .env.example as its .env, as CI does.

The .sqlx check migrates a throwaway judgebot_check database on your server, never your development one.

  • pre-commit checks the staged tree. It runs formatting and clippy when Rust or compiled-in data is staged, Biome and the build for web/, and Biome, astro check, the build and a link check for the docs site. It always runs the repository lints: cargo deny (against the advisory database already fetched, so it works offline), cargo machete, taplo, typos, shellcheck, actionlint, hadolint, a check that the icon’s copies match, a check that every Rust and Node version agrees with the Dockerfile’s, and the compose file. It takes seconds unless clippy has a lot to recompile.
  • pre-push checks the tip of each ref being pushed against the remote’s commit. It runs the repository lints, and, when Rust, compiled-in data, eval/ or the compose file changed, cargo test, the migrations and the .sqlx freshness check (fmt, clippy, web and site already ran at commit). These need the database up (docker compose up -d db) and sqlx-cli. A change to scripts/check.sh, scripts/tools.sh or a workflow, a new branch, or a remote commit your clone has not fetched gets every group. CI runs the same script. Commits that skipped pre-commit (--no-verify, or replayed by a rebase or cherry-pick) get their fmt, clippy, web and site checks only in CI, so run scripts/check.sh before pushing after a rebase.

Other ways to run the gates and linters:

  • scripts/check.sh rust test runs chosen groups on the working tree (any of rust sqlx test web site lint). scripts/check.sh --at <rev> runs them on a commit.
  • sqlx and test wait for any other sqlx or test run on the same Postgres server. #[sqlx::test] names its throwaway databases after each test’s path, so two cargo test runs on one server at once break each other: while a hook may be testing, run the suite as scripts/check.sh test.
  • git commit --no-verify / git push --no-verify skip a hook once.
  • npm --prefix web run fix (or site) applies Biome’s formatting and safe fixes. .tools/bin/taplo fmt formats the TOML.
  • scripts/clean-target.sh [--days N] [--dry-run] reclaims target/: it deletes target/debug/incremental and, with cargo install cargo-sweep, artifacts older than N days (default 14). Run it weekly and after a toolchain bump, a profile change or a large dependency update. scripts/check.sh prints a note when target/ passes 100 GB.

The linters are prebuilt binaries at the versions in scripts/tools.sh, which CI installs too. They live in .tools/ (gitignored). Bumping a version there is the whole upgrade.

Workspace lints deny unwrap, expect, indexing and panic! in every crate, and warn on missing docs and the pedantic group. Reach for a type or a Result instead of a suppression. When one is right, keep it on the one item as #[expect(lint, reason = "…")]. A bare #[allow] is a build error, and an expect that stops matching anything fails the build too, so a stale suppression cannot linger.

deny.toml is the dependency policy. It covers RustSec advisories, the licences an AGPL image may carry, and crates.io as the only source. A dependency with a new licence fails CI until the licence is added there. An accepted advisory is listed there with its reason.

SQL changes. Queries are checked at compile time against the schema. After editing any sqlx::query! or migration, regenerate the committed offline data and include it in the same commit. CI fails when it is stale.

Terminal window
cargo sqlx prepare --workspace -- --all-targets # needs the database up and migrated
SQLX_OFFLINE=true cargo build --workspace # must pass without a database

Install the tool with cargo install sqlx-cli --no-default-features --features postgres,rustls if you do not have it.

Within a major version, migrations only add. A deploy starts the new container, which migrates, while the old one is still answering (scripts/deploy.sh, docs/DECISIONS.md D29), and a judge-cli or judge-mcp of an older minor keeps working against the new schema (judge_core::release). So every earlier release of the same major must still work against it:

  • Fine in a minor or patch release: a new table; a new column that is nullable or has a default; a new index; a new constraint that earlier releases’ writes already satisfy. A rename or a type change is a new column the release writes beside the old one (or reads with a fallback), and the old one stays.
  • Only in a major release: dropping or renaming a column or table, changing a column’s type, making a column NOT NULL without a default, and rewriting rows that earlier releases read or write (moving stored citations to a new key, say). This is also where the old shapes the minors left behind are dropped. Say what changed under the release’s ### Upgrading in CHANGELOG.md.

A major release’s deploy stops the old container before the new one migrates, so it has a short downtime. Its migrations must still finish quickly, because they run inside that window: a slow backfill belongs in a background job.

A released migration is never edited.

Prompt and schema changes. Two tests pin the bytes the bot sends. If you edit a prompt template under crates/bot/src/prompts/ or a struct that feeds the model’s output schema, both fail until you update them:

  • crates/bot/tests/anthropic_golden.rs compares four request bodies to fixtures. Run it with UPDATE_GOLDEN=1 to re-capture them, then read the diff of the fixtures as part of your review.
  • harness_tests in crates/bot/src/synth.rs pins the SHA-256 of the tuned synthesis system prompt. Update the digest in the test when the change is intended.

Taxonomy changes. data/categories.yaml generates the Category enum at build time (crates/core/build.rs), so editing it is a recompile, and every match over the enum has to be updated. data/aliases.yaml and data/notes.yaml are compiled in as data and loaded into the database by judgebot ingest. A deployment that loaded the built-in copy picks up an edit on its first refresh after the upgrade. A change to their entries, not only comments, also copies the previous file into data/legacy/ and appends it to LEGACY in crates/bot/src/ingest/aliases.rs or notes.rs, so a database loaded before the source record existed is still recognised as built-in.

Gold set changes. eval/gold.yaml quotes every rule id (unquoted 702.10 is a float and is rejected). cargo run -p judge-eval -- recall is the free retrieval gate. rescore re-scores stored runs after a gold edit without spending anything.

Model and effort comparisons. A comparison is a plan file in eval/sweeps/ run with judge-eval sweep, not a series of hand-run answer commands: the plan names the models, efforts and repeats, the sweep grades every run with one model and averages them in summary.md. --dry-run first.

The full statement is docs/DECISIONS.md D1. These are the ones a change is most likely to break.

  • Make the bad state unrepresentable. Invariants live in types: closed enums, validated newtypes, NonEmpty, and typestates such as Verdict<Unvalidated> → Verdict<Validated> and the synthesis loop’s single tool round. Prefer that over a runtime check, and a runtime check over a test.
  • crates/core has no I/O. No reqwest, sqlx, tokio-net or provider SDK may be added to it. That dependency fence stands in for effect tracking.
  • Every model call is metered. The only way to send a request is through judge_llm::Metered, which reserves the worst-case cost against JUDGE_MAX_USD first, or, for an embedding, judge_embed::MeteredEmbedder on the same meter. Do not add a second path to a provider.
  • Citations are validated. A quote must be a substring of its source in the retrieved context. Only a Verdict<Validated> can be persisted or shown.
  • The resolver never guesses. Ambiguity in a card name becomes a “did you mean” prompt, not a best effort.
  • Secrets are ApiKey. Anything that authenticates (provider keys, the Discord token, the MCP bearer token) is wrapped so a Debug of any config prints <redacted>. A user’s question text does not go to logs above debug. The card names extracted from it do.

Every user-facing string lives in a catalog, one value per language (docs/DECISIONS.md D35). A new message is a field on its surface’s Messages, with a value in every catalog of that surface, the stub’s (STUB: its own name in ⟦…⟧ with its values) included, and a line in that surface’s every_locale_renders_every_message test. Diagnostics (logs, the command lines’ errors, health checks), JSON output, the model prompts and the agent tools’ text stay English; the replies an interface sends a person, errors included, are catalog text.

Adding a language:

  1. The locale. In crates/core/src/locale.rs, add the Locale variant, its id() (the language subtag, e.g. es), its place in ALL, and its parameter and arm in Catalogs::new and Catalogs::get. Every surface’s Catalogs::new call then stops compiling until it names the new catalog.
  2. The Rust catalogs. Write one Messages value per surface; the compiler lists every missing message:
    • judge_core::locale (the source offer, the contact, the freshness lines);
    • judge_bot::discord::catalog, and the language’s Discord locale tags in TAGS (its command descriptions register under them);
    • judge_bot::alert::catalog (the alert webhook);
    • judge_api::catalog (the HTTP API’s messages and citation labels);
    • judge_agent::catalog and crates/judgebot/src/catalog.rs (the usages);
    • judge_configure::catalog, and the schema and .env help overrides beside it.
  3. The pages. Add the language to LOCALES and a catalog to CATALOGS in web/src/messages.ts and web/src/admin/messages.ts (tsc lists what is missing), an object to judge-config’s page catalog, crates/configure/ui/messages.js, in the format its header describes (cargo test -p judge-configure holds its keys to English’s), and to the docs site: the locale in site/src/locales.mjs, its strings in site/src/i18n.ts, sidebar label translations in site/astro.config.mjs, and its pages under site/src/content/docs/<locale>/ (authored, or manifest entries with locale).
  4. Limits. scripts/check.sh runs the tests that hold every locale to Discord’s limits (100-character command descriptions, the rating form’s 45 and 100) and the const assertion that /judge’s question still fits a “did you mean?” prompt (render::PICK_QUESTION_LIMIT, the least over every catalog): a wordier lead shortens the question a prompt can hold, and the build says so.

The cheapest way to reproduce a bad ruling is the agent session. It runs the bot’s extraction, retrieval and citation validation with you as the model, and it spends nothing. Build judge-cli (cargo build --release -p judge-agent), bring up the database, and step through judge-cli begin | extract | rules | verdict as .claude/skills/judge/SKILL.md describes. judge-cli card, search, get-rules and glossary query the database directly. Use judge-cli judge or a live bot only when the deployed surface is in question, because those calls cost money.

  • Keep a pull request to one change, with the reason in the description. Tests and the .sqlx or golden fixtures it needs go in the same commit as the code.
  • The changelog is CHANGELOG.md, Keep-a-Changelog style. Add a line under Unreleased for anything an operator or user would notice.
  • Card names, rules text and rulings in tests should be short excerpts. Do not commit data dumps, .env files, judge.toml or anything under eval/runs/.

A release is a GitHub release whose tag is vX.Y.Z on a commit of main.

Between releases, main carries the next version with a -dev suffix (2.0.0-dev), set in the same places as step 1 below when work toward that release starts. Each migration records the release and commit that ran it in instance_release, and judge-cli, judge-mcp and judgebot itself compare their own release with it (judge_core::release):

  • Within a major (2.1.0-dev included), migrations only add, so builds work together in either direction. An older tool prints a note, and an older judgebot (a rollback) starts.
  • A new major’s pre-release (3.0.0-dev, 3.0.0-rc.1) is where its breaking migrations are made, so it works only with the build that migrated the database. While main is on one, switching a checkout back to an earlier commit means a fresh development database (or a backup of one): judgebot refuses the one a later build migrated.
  • Across a major, or against a database missing one of their migrations, the tools stop and judgebot refuses to start, each saying what to update.
  1. Move the Unreleased changelog section under a ## [X.Y.Z] heading, and set that version in [workspace.package] in the root Cargo.toml (then cargo update --workspace for Cargo.lock), in web/ and site/‘s package.json and package-lock.json, and in the image-tag examples in docs/DEPLOYMENT.md and the configuration page. The binaries report the crate version (the MCP server, the ingest steps’ User-Agent). scripts/release-version.sh checks the copies: check.sh lint that they agree, and publish-image.yml that they are the tag’s and the changelog has its heading. A release that fails it publishes no image: delete the release and its tag, fix the version, and create both again.
  2. Wait for the push’s Publish image run to finish. Otherwise the release and that run build in parallel.
  3. Create the release (gh release create vX.Y.Z --generate-notes, or the web form).

Publishing it runs publish-image.yml. That tags the image already built for the commit as X.Y.Z, X.Y and, from 1.0 on, X, without rebuilding. So the release is the same image that has been running as latest, down to the platform digests. A commit with no image of its own (a docs-only commit, or a build a later push cancelled) is built first, through CI. Pre-release tags such as v1.0.0-rc.1 get only their exact version tag, and need that exact version in the tree (step 1) first, because the binaries report it and a new major’s pre-release works only with its own build. latest keeps following main.

The repository carries a .mcp.json and a judge skill under .claude/skills/. With Claude Code, they let Claude act as the model in the judge pipeline through judge-mcp. Other tools ignore them. CLAUDE.md is Claude’s working notes for this codebase. It is kept in sync with the docs, but the docs are the source of truth.