Skip to content

Model providers

The reference for how the judge reaches its models. It covers the provider-neutral seam (crates/llm), the two chat backends (crates/anthropic, crates/openai), the two embedding backends (crates/embed), and the judge.toml file that chooses among them. The seam keeps the pipeline, the prompts, the validation and the typestates independent of the provider. docs/DECISIONS.md (D6–D9) records the reasoning.

Goal. An operator who is not the author can run the judge with:

  • Anthropic models through the first-party API, a proxy that speaks the Messages API (LiteLLM’s /v1/messages), or a cloud: Claude Platform on AWS, Amazon Bedrock or Google Vertex AI. The clouds use the Messages API shape with their own auth and hostnames.
  • Any model behind an OpenAI-compatible chat completions endpoint: LiteLLM, OpenRouter, Ollama, vLLM, llama.cpp, Azure OpenAI, Bedrock’s and Vertex’s OpenAI-compatible endpoints, OpenAI itself.
  • Embeddings from Voyage or any OpenAI-compatible /v1/embeddings endpoint.
  • A different model per stage: a cheap model for extraction, a strong one for synthesis, possibly on different providers.
  • A spend cap that still holds when the price table does not know the model.

Non-goals.

  • Streaming. Every request keeps max_tokens ≤ 16k, inside the non-streaming guidance.
  • A plugin system. Providers are Rust crates in this workspace, chosen by configuration.
  • Every quirk of every OpenAI-compatible server. The OpenAI backend has a small set of dialect settings and otherwise follows OpenAI’s documented API.
  • The Responses API (/v1/responses). A Responses-only model would be a third backend, not an option.

The pipeline sends four request shapes, all non-streaming. serde decodes each answer into the type its schema is generated from (invariant I5 in docs/DECISIONS.md).

# Call System Tools Tool choice Structured output Effort
1 Extraction cached, stable none — Extraction schema measured per model (EXTRACT_EFFORTS), else low
2 Synthesis, first turn cached, stable lookup_rules, strict auto, ≤1 call Verdict schema measured per model (SYNTH_EFFORTS), else high; a truncated answer reruns at medium, or low from medium
3 Synthesis, tool continuation same same same (cache-preserving) same same
4 Synthesis, citation retry same lookup_rules still listed none same same

The pipeline also needs:

  • Refusal detection (stop_reason: refusal → JudgeError::LlmRefused).
  • Truncation detection (max_tokens → Truncated, retried once at lower effort).
  • Usage for the spend cap.
  • The model’s own assistant turn echoed back verbatim in call 3. That is thinking blocks with signatures on Anthropic, reasoning_content on DeepSeek-style servers and tool_calls on OpenAI.

Embeddings need embed(texts, kind) -> Vec<Vec<f32>> at a fixed width.

The judge’s guarantees are enforced client-side. Citations are validated against Context (I3), the typestate bounds the tool round (I4), and decoding enforces the schema (I5). Structured-output enforcement on the server is an optimisation that lowers the retry rate, so a backend without it makes more retries and keeps the same guarantees. Providers are therefore configurable without touching core.

crates/bot/tests/anthropic_golden.rs pins the four Anthropic request shapes byte-for-byte against captured fixtures. UPDATE_GOLDEN=1 re-captures them after an intended prompt or schema change. Review the diff.

core ← llm ← { anthropic, openai } (chat backends)
core ← embed { voyage, openai-embeddings } (embedding backends)
bot ← llm, embed (adapters + build_deps; picks backends by config)

crates/llm (judge-llm) holds:

  • The neutral types and the port.
  • The one-tool-round typestate (Synth<Fresh | ToolRequested | Final>).
  • classify (refusal / truncation / tool / verdict over a neutral response).
  • The spend cap.
  • The shared HTTP retry loop (408/409/429/5xx honouring retry-after, identical for every backend).

crates/anthropic and crates/openai are its backends. Each owns its wire types and its schema-subset transform. crates/bot’s extract.rs and synth.rs (prompts, rendering, budget, hydration, truncation retry) depend on judge-llm only. crates/core depends on none of it.

// crates/llm: what providers implement …
#[async_trait]
pub trait Backend: Send + Sync {
async fn complete(&self, req: &ChatRequest) -> Result<ChatResponse, LlmError>;
fn capabilities(&self) -> Capabilities;
fn model(&self) -> &str; // as the backend will bill it
fn provider(&self) -> &'static str; // the price table and the log line key on it
}
// … and what the pipeline calls. Sealed: `Metered<B>` is its only implementation,
// so every send is behind the spend cap by type, not by composition discipline.
pub trait ChatModel: sealed::Sealed + Send + Sync { /* complete, capabilities, model */ }
pub struct ChatRequest {
pub max_tokens: u32,
pub system: Vec<TextBlock>, // text + CacheHint
pub turns: Vec<Turn>,
pub tools: Vec<ToolSpec>, // name, description, JSON Schema, strict
pub tool_choice: ToolChoice, // Auto { parallel: bool } | None
pub output: Option<OutputSchema>, // the JSON Schema of the type we decode into
pub effort: Option<Effort>, // Low | Medium | High | XHigh | Max
pub thinking: bool, // extended reasoning where a backend offers it as an opt-in
pub fallbacks: Option<RefusalFallback>, // Default | Models(Vec<String>): server-side refusal fallback
} // (both carried so the Anthropic bodies stay byte-identical)
pub enum Turn {
User(Vec<TextBlock>),
/// The model's own previous turn, replayed byte-for-byte. Opaque on purpose:
/// thinking signatures, reasoning_content, tool_calls — whatever the backend
/// returned is what it gets back. Only the backend that produced it reads it.
Assistant(AssistantTurn), // { backend: &'static str, raw: serde_json::Value }
ToolResults(Vec<ToolResult>), // { call_id, content: String, is_error }
}
pub struct ChatResponse {
pub text: Vec<String>, // text blocks in order; the JSON is the last
pub tool_calls: Vec<ToolCall>, // { id, name, input: Value }
pub stop: Stop, // EndTurn | MaxTokens | ToolUse | Refusal(details) | Other(String)
pub usage: Option<Usage>, // input, output, cache_read, cache_write; None when unreported
pub model: String, // as billed (may differ under fallbacks)
pub assistant: AssistantTurn, // for replay
}
  • AssistantTurn is opaque. The neutral layer never inspects it. It exists so the continuation request can replay it. A backend receiving an AssistantTurn tagged with another backend’s name returns LlmError::ForeignTurn rather than guessing.
  • CacheHint is a hint. Anthropic emits cache_control. OpenAI ignores it and caches by prefix on its own. LiteLLM forwards cache_control when the upstream is Anthropic. Placement rules (stable content first) are left to the adapters.
  • OutputSchema carries the schemars schema untransformed. Each backend applies its own subset transform. Anthropic’s sets additionalProperties:false, rewrites oneOf→anyOf and strips constraints outside the subset. OpenAI strict mode’s additionally lists every property in required, turns optional fields into anyOf [T, null], and drops format and default. Both are pure functions with walk-and-assert tests. Decoding is unchanged, so a stripped constraint is still enforced client-side.
  • Property order is the order the model writes. A backend that enforces the schema makes the model emit keys in the schema’s order. schemars therefore keeps declaration order (preserve_order), and Citation moves its kind tag, which schemars appends, to the front, so the schema asks for kind, then the reference, then quote, as the prompt does. With alphabetical keys, Opus 5.5 wrote a citation’s values one key over.
  • Effort maps per backend. Anthropic gets output_config.effort. OpenAI gets reasoning_effort (low|medium|high, xhigh|max → high), and only when the provider’s reasoning_effort = true. Asking for effort on a provider that cannot send it is a load error, not a silent drop. A stage that names no effort gets the effort the model’s gold runs chose for it: for extraction judge_llm::EXTRACT_EFFORTS (claude-opus-5-5 medium, claude-sonnet-5-5 low, claude-haiku-5-5 high), else low; for synthesis judge_llm::SYNTH_EFFORTS (claude-opus-5-5 medium, claude-sonnet-5-5 low, claude-haiku-5-5 medium), else high. A truncated answer is rerun once at medium, or at low from medium (Effort::truncation_rerun); one at low is not rerun. Ids match exactly, so a Bedrock anthropic.claude-opus-5-5 is unlisted and runs at the unlisted efforts unless its stage says otherwise.
  • Capabilities = { structured_output: Enforced | JsonMode | PromptOnly, strict_tools, effort, cache_hints, refusal_fallbacks }. The adapters use it for two things:
    • When enforcement is PromptOnly/JsonMode, they append the schema to the user turn (judge_llm::schema_block), so the pinned system-prompt digest holds on every backend.
    • They log what they are relying on.

Metered<B> wraps a Backend. It reserves the worst case, sends, then settles on the usage the response reports. The reservation is sized from the serialized neutral request (a few percent larger than the wire body, pessimistic either way). Price is a closed sum:

  • Table: the built-in table (judge_llm::PRICES, Anthropic first-party models). It is re-read for the model the response names, because an Anthropic fallback may route elsewhere. An entry is a Rate: one card, or two where Anthropic bills a longer prompt at a dearer card (claude-haiku-5-5 past 100K prompt tokens, cache reads and writes counted). The reservation picks the card at half as many tokens again as its estimate, so a newer tokenizer’s higher count does not settle a request five times over it.
  • PerToken: the operator’s rate from [models.<stage>.pricing], settled at that rate whatever model the response names.
  • Free: never reserves, still counts calls (pricing = "free" on a provider).

Around the meter:

  • Each process has one SpendMeter, shared by both stages and by every clone. JUDGE_MAX_USD is the cap.
  • A 2xx body that fails to decode is still billed: each backend exposes a lenient usage_of(body).
  • A response with no usage (an OpenAI-compatible server may leave it out) keeps the whole reservation as its cost, as an embedding does.
  • ChatModel::complete takes a Bill: the run’s judge_core::Tally and the Stage the call is for. Settling adds a Charge to it, so a run’s cost is its own however many runs share the meter (judge_runs, D33). A response with no usage is charged at the reservation it kept, as Cost::WorstCase. A run whose caller went away (Tally::abandon) gets LlmError::Abandoned before anything is reserved or sent.
  • Each call logs an llm call line with a provider field.
  • Models::{single, pair, priced} take the meter and bare backends and meter them themselves. The fields are private, so there is no uncapped model and no foreign meter.
  • Embeddings bill to the same meter (§4.3). SpendMeter::reserve_embedding_usd hands out the Reservation they settle, or release when the request was never billed. Dropped unsettled (a cancelled request the provider may have billed) it keeps the worst case, as a cancelled chat call does. Embedding requests and refusals have their own counters (embedding_calls, embedding_refusals), so calls(), the ledger’s calls and the questions-refused alert stay chat-only.

One wire format serves several endpoints. The body is the Messages API everywhere, and the endpoints differ in auth, URL and a feature mask. Endpoint is an enum, so a new endpoint is an exhaustive-match compile error, not a config typo. The formats were verified against the live docs 2026-09-02.

Endpoint Auth URL Model id Mask
Direct { base_url, api_key } x-api-key {base}/v1/messages claude-opus-5-5 none
Proxy { base_url, api_key, header } x-api-key or Authorization: Bearer {base}/v1/messages whatever the proxy routes fallbacks off
ClaudePlatformOnAws { base_url, region, workspace_id, credentials } SigV4, service aws-external-anthropic, header anthropic-workspace-id https://aws-external-anthropic.{region}.api.aws/v1/messages bare fallbacks off
Bedrock { base_url, region, credentials } SigV4, service bedrock-mantle https://bedrock-mantle.{region}.api.aws/anthropic/v1/messages anthropic.-prefixed (a whole anthropic segment is required) fallbacks, output_config.format, tool strict and every anthropic-beta off. The schema goes in the prompt.
Vertex { base_url, project, region, token } GCP ADC bearer token model in the URL, anthropic_version in the body (wire::ModelField). The origin depends on global, a multi-region or a specific region bare fallbacks off

Every endpoint’s base_url is derived from the region (or project) and can be overridden in the provider table, so a changed platform hostname is a config edit, not a release.

Structured outputs, strict tools, adaptive thinking/effort and prompt caching are GA on Vertex. The Bedrock endpoint is Bedrock’s Messages-shaped API (bedrock-mantle), where Bedrock documents structured outputs as unsupported, hence the mask in the table. Anthropic documents server-side fallbacks for the Claude API only (Claude Sonnet 5.5’s migration guide says “Claude API only”; the refusals page names Bedrock, Google Cloud and Foundry as unsupported). Claude Haiku 5.5 has no server-side fallback on any endpoint: the "default" form the pipeline sends is accepted, and a declined request stays declined. Every endpoint but direct therefore turns SynthConfig::fallbacks off with a warning rather than sending a beta the endpoint may reject. A provider table’s refusal_fallbacks = true | false overrides the endpoint’s default either way: true for a proxy that passes the beta through, or for Claude Platform on AWS once it is known to take it; false to never send it. true on bedrock is a load error, because that endpoint drops every anthropic-beta header and the field cannot travel without its beta.

Each provider table resolves to one Endpoint, shared by the stages that name it. For the cloud endpoints:

  • They sit behind judge-anthropic’s aws (aws-config + aws-sigv4) and gcp (gcp_auth) Cargo features. Both are on by default, forwarded from judge-bot’s own features and named in the Dockerfile. A lean build cannot name these endpoints, and the loader says “not built”.
  • Credentials come from the platforms’ standard chains (env, profile, instance role / ADC), never from judge.toml. The cloud endpoints do not accept API keys.
  • The chains are lazy, so loading never touches the network. Config::probe_auth resolves each endpoint once at startup. An empty chain fails there, naming the provider and the endpoint, not on every question.

crates/openai is hand-written like the Anthropic client. It owns its wire types and uses no SDK.

Neutral Chat completions
system blocks one system message: the texts joined, or content parts each carrying its cache_control when cache_hints = true (which LiteLLM honours for Anthropic upstreams)
Turn::User user message
Turn::Assistant(raw) the choices[0].message object replayed verbatim (keeps tool_calls, reasoning_content, anything else)
Turn::ToolResults one tool message per result with tool_call_id
ToolSpec tools: [{type: function, function: {name, description, parameters, strict}}]
ToolChoice::Auto { parallel: false } tool_choice: "auto", parallel_tool_calls: false
ToolChoice::None tool_choice: "none"
OutputSchema response_format: {type: json_schema, json_schema: {name, schema, strict: true}}, or {type: json_object} + schema in the user turn, or prompt only. The structured_output option chooses.
Effort reasoning_effort when reasoning_effort = true
max_tokens max_completion_tokens (OpenAI) or max_tokens (most compatible servers), by the max_tokens_param option
stop finish_reason: stop→EndTurn, length→MaxTokens, tool_calls→ToolUse, content_filter→Refusal. A non-null message.refusal is also a Refusal.
usage prompt_tokens − prompt_tokens_details.cached_tokens → input, cached_tokens → cache_read, completion_tokens → output
tool call arguments function.arguments is a string. It is parsed with serde, then validated against LookupRulesInput as on Anthropic.

Dialect settings live on the provider table (§5) with defaults that are right for OpenAI and LiteLLM. auth = "bearer" | "api-key" covers Azure’s header. base_url can carry Azure’s ?api-version= query.

judge-embed has VoyageEmbedder and OpenAiEmbedder. The latter sends POST /v1/embeddings with input, model and, unless send_dimensions = false, dimensions. It ignores InputKind, because that API has no query/document distinction. [models.embed] chooses between them. Without one, VOYAGE_API_KEY selects Voyage and a blank key turns the vector search off.

Spend cap. Both adapters implement the open EmbedBackend and report each request’s usage (Voyage’s usage.total_tokens; OpenAI-compatible usage.total_tokens, else prompt_tokens). Only MeteredEmbedder<B> implements the sealed WithSpace that everything downstream takes, and Config::embedder/vectors build it on the process’s SpendMeter, the one the chat stages bill to. Per request:

  • It reserves the worst case: every text’s UTF-8 bytes plus 16 tokens each, at the model’s price. No tokenizer the providers use makes a token of less than a byte, and the 16 cover Voyage’s input_type prompt and the start and end markers. A request the cap cannot fit is refused (SpendCapExceeded) before it is sent. A batch is one request.
  • It settles on the reported usage. A 2xx with no usage, or 0 tokens for texts it was sent, keeps the whole reservation. A 2xx whose body fails afterwards (a vector short, the wrong width) is still billed. A request whose future is dropped keeps the reservation; one that failed before a 2xx releases it.
  • It logs an embedding call line and counts the call.

EmbedPrice mirrors Price, in USD per million input tokens:

  • Table: Voyage’s list price (judge_embed::VOYAGE_PRICES, from docs.voyageai.com/docs/pricing, checked 2026-10-09), from the first token. Voyage’s free allowance is per account and not subtracted. An unknown Voyage model prices as the dearest listed, voyage-3-large’s $0.18.
  • PerToken: [models.embed.pricing] input = …, which overrides the list on Voyage. A price of 0 reserves nothing but, as for a chat stage, is still refused once the cap is exhausted; only pricing = "free" on the provider is never refused.
  • Free: pricing = "free" on an openai provider. Calls are counted, never refused.

An openai-kind embedding model has no list. It needs [models.embed.pricing] or a free provider, else the file does not load (models.embed: no price for …).

At the cap the callers degrade rather than fail. A refused query embedding leaves retrieval without its vector source (WARN once per streak of refusals). A refused call embedding stores the call with a NULL vector for the next ingest embed. ingest embed, reembed --yes and the refresh’s embed step fail naming the cap and keep what they embedded, and the scheduler alerts on the failed step. reembed --yes refuses a switch whose worst case does not fit what is left before it clears anything.

Vector space identity. Vectors from two models cannot share a column, and pgvector’s HNSW index needs a fixed width, so every embedder carries a Space (WithSpace::space): provider kind (voyage | openai, not the operator’s table name), model and dimensions. Space::check (pure, in judge_embed::space) is the only definition of “same space”.

  • The one-row table embedding_space records what the stored vectors are. Migration 20260904000001 creates it and seeds voyage/voyage-3.5/1024 for a database that already held vectors. ingest embed writes the row with the first vector it writes, never before. It refuses on a mismatch, or when the columns’ actual vector(N) typmod differs (db/space.rs column_width). It never relabels vectors it did not write.
  • The adapters hold no bare Embedder. PgRetriever, PgLibrary and PgCallStore take Arc<db::Vectors> (Config::vectors(pool), one per process), which embeds nothing until the stored space equals its own. The row is re-read on every use, and once at startup so the verdict sits beside the config summary. A running bot therefore picks up the first ingest embed. A reembed under it turns vector search off, with an error-level log naming both spaces, instead of erroring or mixing.
  • Writers hold the space. PgCallStore::persist and every ingest embed batch take the shared side of CALLS_REWRITE_LOCK in their transaction and read the row under it (hold_space / Vectors::hold). switch_space takes the exclusive side, as the CR loader and the retirement pass do. A switch therefore waits for in-flight writes, and a write after it sees the new row.
  • ingest reembed --yes (switch_space) is the only thing that changes the row and the column width.
    • It first probes the embedder with one short text, so a wrong key, URL, model or width fails with the old vectors intact.
    • In one transaction it retypes the vector columns, drops and recreates the HNSW indexes (definitions in VECTOR_TABLES, verbatim from the migrations), NULLs every vector and rewrites embedding_space. Then it runs the normal embed loop.
    • When the database already holds the configured space, it only fills NULL rows. That is idempotent, and it is how an interrupted refill resumes.
    • --clear re-pays every row in the same space.
    • Without --yes it prints the row counts and bytes, the estimated cost at the configured price and its worst case (as the cap reserves it), and what is left under the spend cap (with a JUDGE_MAX_USD that fits the run when it does not), exits non-zero and changes nothing.
  • config::Dimensions is 1..=2000 (HNSW’s limit) at load, and VOYAGE_DIMENSIONS has the same bound. The migrations create vector(1024), and reembed is how it changes.

Default setup. With no judge.toml, the binaries build their setup from .env: Anthropic direct with ANTHROPIC_API_KEY, claude-opus-5-5 for both stages, Voyage if VOYAGE_API_KEY is set. The eval numbers were measured on that setup.

A judge.toml (path from JUDGE_CONFIG, else ./judge.toml if present) selects providers and models. judge.example.toml documents every option with its default. Two tests in config.rs pin it: one loads it as shipped, the other with every commented table uncommented and each endpoint named by a stage, so a renamed option fails the tests.

# Secrets are named by environment variable, never written here.
[providers.anthropic]
kind = "anthropic" # anthropic | openai
endpoint = "direct" # direct | proxy | claude-platform-on-aws | bedrock | vertex
api_key_env = "ANTHROPIC_API_KEY"
# refusal_fallbacks = true # default: true on direct only
[providers.litellm]
kind = "openai"
base_url = "http://litellm:4000/v1"
api_key_env = "LITELLM_KEY"
# dialect settings, all optional, shown with defaults
structured_output = "json_schema" # json_schema | json_object | prompt
strict_tools = true
reasoning_effort = false
max_tokens_param = "max_tokens" # max_tokens | max_completion_tokens
cache_hints = false
[providers.ollama]
kind = "openai"
base_url = "http://ollama:11434/v1"
structured_output = "json_object"
pricing = "free" # local: the spend cap never trips on this provider
[models.extract]
provider = "ollama"
model = "qwen3:8b"
max_tokens = 2000
[models.synth]
provider = "anthropic"
model = "claude-opus-5-5"
effort = "medium"
max_tokens = 16000
# USD per million tokens. Required when the model is not in the built-in table,
# unless the provider is `pricing = "free"`. A model the cap cannot price is a
# startup error, not a silent under-estimate.
[models.synth.pricing]
input = 4.0
output = 20.0
cache_read = 0.2 # defaults to `input`
cache_write = 5.0 # defaults to 1.25 × `input`, Anthropic's write premium
[models.embed]
provider = "voyage" # kind = "voyage" provider, implied when absent
model = "voyage-3.5"
dimensions = 1024
# [models.embed.pricing] # USD per million tokens: required on an `openai`
# input = 0.02 # provider not marked free; overrides Voyage's list

The loader enforces these rules at load time, each with a message naming the key:

  • The structs are typed with deny_unknown_fields, so a typo is an error. nutype validators cover BaseUrl (absolute http(s) with a host), Region, Project, WorkspaceId (the shapes the platforms document) and Dimensions.
  • Every api_key_env a stage names must be present and non-blank. It is read once into a redacted ApiKey. A table no stage names is parsed, and its key is never read.
  • An option that would be silently ignored is an error naming both keys: auth without api_key_env, effort on an openai provider with reasoning_effort = false, a stage price on a pricing = "free" provider, a cloud-endpoint key on a keyed endpoint.
  • A model on an openai provider must be priced or its provider pricing = "free", an embedding model included ([models.embed.pricing]). The built-in tables price unknown Anthropic chat models (as the default model, so a dearer one is under-counted unless it has a price of its own) and unknown Voyage embedding models (as the dearest listed). An unknown OpenAI-compatible model could be anything.

The loader is hermetic: from_toml/from_vars read JUDGE_MAX_USD and the keys through an injected environment, so its tests never touch the process environment. Every binary logs Config::summary() at startup (config=<file or "env"> extract=ollama/qwen3:8b synth=anthropic/claude-opus-5-5 embed=voyage/voyage-3.5 cap=$5.00). judge-cli config prints the same, secrets redacted.

Under Docker. JUDGE_CONFIG in .env is a host path (./judge.toml, as cargo run reads it). docker-compose.yml bind-mounts that file into judgebot and refresh at /etc/judgebot/judge.toml and sets the containers’ JUDGE_CONFIG to that path (${JUDGE_CONFIG:+…}), so one variable serves cargo run and compose alike. This has two consequences:

  • A ./judge.toml with JUDGE_CONFIG blank is read by cargo run and by nothing under Docker. The containers see only the mounted file, and blank mounts the tracked example, which nothing reads.
  • Editing the mounted file’s content is not a change up -d recreates for, so run scripts/deploy.sh: its new container reads the file, with no downtime.

judge.toml is gitignored (per host, not secret). So is docker-compose.override.yml, the documented home for cloud-credential mounts. refresh resolves all three stages of the file as the others do, so every named provider’s key must be in .env for it as well.

  • crates/core: Deps, judge(), Verdict::validate, Citation. No provider type reaches it.
  • The synthesis and extraction prompts, the Harness enum and the pinned SHA-256 of the system prompt. An OpenAI backend gets the same Harness::Tool prompt. The tool is still named lookup_rules, and the wire format is left to the backend.
  • The agent surface (crates/agent, sessions, MCP, CLI, the skill). The calling agent is the model there, so providers do not apply. The built-in judge tool reports which model it is running in its replies, nothing more.
  • Retrieval, ratings, retirement, renumbering, the HTTP API, the web app, deployment.

judge-eval answer --config <judge.toml> runs the gold set on another provider and stores {provider, model} per stage in the run file, so show and rescore can compare two providers on the same questions. The recall gate is unaffected: it uses no chat model, and --vectors uses the configured embedder. A cheap smoke target for the OpenAI backend is answer --limit 3 against a local Ollama model with pricing = "free". It checks that the backend still round-trips a tool call and a verdict, not quality.