Prompt & Model Benchmark Lab

The Problem

Picking an LLM for a task is mostly vibes. Teams default to the biggest model for everything, or the cheapest, and never actually measure which one is right for this job. My own earlier tool, Prompt Test Lab, ran A/B tests — but its “scoring” was fake: keyword overlap and a hardcoded accuracy = 0.75. A benchmark you can’t trust is worse than none; it launders a guess as a number.

What I Built

Part of the verification-gated agent-work pattern (see /toolkit/patterns/verification-gated-agents) — here: judge composites are recomputed in code from per-dimension scores (never self-set by the judge), and any cell where the judge model is also the candidate is flagged self-graded and stripped of confidence before it can count as evidence.

Part of the design-of-experiments optimization pattern (see /toolkit/patterns/design-of-experiments) — here: the test matrix is Cases × Candidates (model + prompt + params) scored under one grading pipeline, feeding a constraint-based /recommend endpoint (cost ceiling, latency ceiling, quality floor) rather than a single fixed winner.

The Benchmark Lab turns “which model should I use?” into a query with evidence behind it. One primitive — a Run = Cases × Candidates (model + prompt + params) — powers both model benchmarks and prompt A/B tests. Every output is graded for real: deterministic graders where ground truth exists, an LLM-as-judge (rubric, temperature 0) where it doesn’t, and every score carries a calibration confidence so you know how much to trust it. A POST /recommend endpoint takes a task plus constraints (cost ceiling, latency ceiling, quality floor) and returns a model, the reason, and its confidence.

The stack is Next.js 16 full-stack with a BullMQ/Redis worker for execution and grading, Prisma 7 on Postgres, multi-tenant from the schema up, with a fail-closed cost gate and OpenTelemetry gen_ai spans on every model and judge call. Candidates can be Anthropic, OpenAI, or Groq-hosted cloud models, or on-device models via Ollama — keyless, zero marginal cost — so you can compare cloud vs local on the same tasks.

Why It Matters

Model choice is a per-task decision, not a global one. A subtle security audit and a bounded CRUD endpoint reward completely different models — and on bounded work, the expensive tier often buys nothing. Without measurement you can’t see that; you just overpay or underperform. This makes the trade-off legible: same tasks, same grading, real numbers, with the confidence attached.

The Model-Fitness Mosaic

The signature view is a mosaic — every model scored by exercise (audit, planning, code-gen, bug-fix, UI), complexity, and stack layer (UI, middleware, database, infra). Filter to a situation and read who wins. Empty cells are honest gaps: “no evidence yet,” not a fabricated zero.

Evidence You Can Drill Into

Every score is clickable. Open a cell and you see each contributing run: the date, how it was graded, the calibration, and — for live runs — the raw prompt, the model’s actual output, and the judge’s verdict and rationale. A one-line Trust banner tells you straight whether the evidence is objective, mixed, or judge-only-and-directional. No number hides its provenance.

Recommendation, Not Just a Leaderboard

The leaderboard makes the evidence inspectable; the recommendation explorer turns it into an answer. Pick a task and priority (speed, quality, balanced), set optional constraints, and it returns the model with a cited reason — quality, cost, latency — plus its confidence and the alternatives it beat.

Two Kinds of Evidence, One Wall Between Them

The benchmark is prospective and calibrated. But real build sessions across my other projects generate their own model-fitness signal — which model shipped a chunk, whether it held. The lab ingests these as field observations, quarantined behind a hard wall: they show up as corroboration, but they can never blend into a calibrated score or move a recommendation’s confidence. Observational evidence stays labeled observational.

Technical Decisions

Grading is the crux, so it’s the part I was strictest about. The judge is gated by calibration and can never set its own composite — that’s computed in code from its per-dimension scores. When the judge model happens to be one of the candidates, the pipeline flags the cell self-graded and strips its confidence, because a model scoring its own work isn’t evidence.

Architecture

A pnpm monorepo split into a Next.js dashboard, a BullMQ worker, and shared core logic: apps/web (Next.js 16, the dashboard + all /api/* routes), apps/worker (the BullMQ consumer that actually runs and grades cells), packages/core (execution, grading, cost-gating, recommend, field/inferred evidence — imported by both apps, so there is one grading implementation, not two), packages/db (Prisma 7 schema, multi-tenant from the Org row down), packages/config (zod-validated env). Postgres 17 is the system of record; Redis 7 backs both the job queue and the cost gate.

flowchart LR
  A["POST /api/runs<br/>Cases × Candidates"] --> B[["Redis: BullMQ 'runs' queue"]]
  B --> C["worker: loadCells<br/>(Postgres: TaskCase, Model, PromptVersion)"]
  C --> D["executeCall<br/>cost gate (Redis Lua) + OTel gen_ai span"]
  D --> E1["OpenAI / Anthropic / Groq<br/>cloud candidates"]
  D --> E2["Ollama, 'local' provider<br/>OpenAI-compat :11434/v1"]
  E1 --> F[("Postgres: Result")]
  E2 --> F
  F --> G{"ground truth?"}
  G -->|yes| H["deterministic grader<br/>exact / regex / json_schema / …"]
  G -->|no| I["LLM-as-judge<br/>Groq Llama 3.3 70B, temp 0"]
  H --> J[("Postgres: Score")]
  I --> J
  J --> K["aggregateBenchmark<br/>windowed"]
  K --> L[("Postgres: BenchmarkScore")]
  L --> M["POST /api/recommend<br/>constraint filter + priority weights"]
  N[("FieldObservation<br/>build-loop / ABC ingest")] -.corroboration only, walled.-> M
  O[("InferredSignal<br/>fills gap pre-calibration")] -.provisional only.-> M
  M --> P["Dashboard: mosaic, leaderboard,<br/>evidence, recommend explorer"]

Models — which, where, why:

Stage	Model / tier	Where it runs	Why there
Candidate under test	Whatever the Run’s grid names — cataloged Anthropic (Claude family), OpenAI (GPT-5.x family), Groq-hosted (Llama 3.1/3.3, gpt-oss 120B/20B), or a local Ollama model via the local provider	apps/worker, one call per grid cell (executeCall)	The object under measurement, not a fixed choice — a Run is Cases × Candidates, so any cataloged model can be a candidate
Judge	Groq llama-3.3-70b-versatile by default (JUDGE_MODEL_ID env override), temperature 0	packages/core/grade/judge.ts, called from the worker’s gradeRunStep after execution	Groq’s inference speed keeps grading cost near-zero next to candidate spend; temperature 0 for determinism; its JSON verdict never sets the composite score directly — that’s always recomputed in code from per-dimension numbers
Deterministic graders	None — rule-based (exact, contains, regex, numeric, json_schema)	packages/core/grade/deterministic.ts	Used whenever a task has ground truth; zero cost, zero latency, zero nondeterminism — the judge only runs where a rule genuinely can’t grade
Local / on-device	Ollama, reached over its OpenAI-compatible endpoint (local provider, OLLAMA_BASE_URL, default http://localhost:11434/v1)	apps/worker, same call path as any cloud candidate	Keyless and zero marginal cost, so a Run can put a local model on the same grid as cloud models and compare them under identical grading

Reasoning models are a known trap: gpt-oss on Groq sometimes returns its answer in the response’s reasoning field instead of content. callModel falls back to reasoning when content is blank — see Lessons.

Tools & infra — which, why:

• PostgreSQL 17 (Prisma 7) — system of record for orgs, models, tasks, runs, results, scores, benchmark scores, judge calibration, field observations, and inferred signals; every table carries orgId, so multi-tenancy is enforced in the schema, not the app layer.
• Redis 7 + BullMQ — the "runs" queue decouples run execution (which can take minutes across a full grid) from the Next.js request/response cycle; the same Redis instance backs the cost gate.
• Cost gate (Redis, Lua script) — an atomic INCRBYFLOAT-style reservation checks current + estimate <= dailyBudget before any call goes out; fails closed on budget-exceeded or kill-switch, fails open only on a Redis transport error, so a Redis outage degrades rather than blocks in-flight runs.
• OpenTelemetry (@opentelemetry/api) — a gen_ai-convention span wraps every model and judge call (gen_ai.system, gen_ai.request.model), giving cost/latency/error tracing without a bespoke logging layer.
• zod (packages/config) — validates DATABASE_URL, REDIS_URL, KEY_ENCRYPTION_SECRET, budget, and provider keys at boot; fails fast on a misconfigured environment instead of failing mid-run.
• In-house key vault (packages/core/vault) — encrypts stored provider API keys at rest with KEY_ENCRYPTION_SECRET, rather than keeping them plaintext in the multi-tenant Model/candidate config.
• Docker Compose — local dev only: postgres:17 and redis:7 containers on non-default ports (5433/6380) so they don’t collide with other local services.

How it works

1. A Run is created as Cases × Candidates — a task’s TaskCase rows crossed with a set of Candidates (model + prompt version + params); expandGrid() produces the Cell grid.
2. POST /api/runs enqueues the run on Redis’s "runs" BullMQ queue; apps/worker’s Worker("runs", processor) picks it up.
3. makeLoadCells resolves each cell against Postgres — the real TaskCase input, the model’s catalog price, and the PromptVersion template — and builds a resolve() closure that renders the final prompt and selects the provider’s API key / base URL per cell.
4. For each cell, executeCall reserves estimated spend in Redis (the cost-gate Lua script, atomic), wraps the call in an OTel gen_ai span, and dispatches through callModel to the right provider adapter — OpenAI and Groq speak OpenAI-compatible chat completions, Anthropic speaks its native Messages API, and local hits an Ollama instance’s OpenAI-compatible endpoint. Reasoning-model output landing in reasoning instead of content is captured via fallback (the bug described in Lessons).
5. Actual cost is computed from the real usage tokens returned by the provider and reconciled against the earlier reservation; the Result (output, tokens, latency, cost, error) is persisted to Postgres.
6. Once execution finishes, gradeRunStep grades every Result: a deterministic grader runs where the task has ground truth (exact/contains/regex/numeric/json_schema); otherwise the LLM-as-judge scores rubric dimensions at temperature 0, with the composite always recomputed in code from those dimensions, never trusted from the model’s own output. A judge grading its own candidate’s output is flagged self-graded and its calibration confidence is stripped before persistence.
7. Scores aggregate into windowed BenchmarkScore rows — quality composite, p50/p95 latency, average cost, sample size, calibration confidence — the only rows the recommend engine is allowed to read.
8. POST /api/recommend loads BenchmarkScore stats, filters out models that violate a cost ceiling / latency ceiling / quality floor, and scores the survivors with priority-weighted coefficients (speed / quality / balanced) to return a model, a cited reason, a confidence, and the runners-up.
9. Field observations — real build-loop sessions, ABC head-to-head comparisons — ingest separately into FieldObservation and attach to the recommend response as labeled corroboration only; they never enter ModelStat, a score, or a confidence value (the evidence wall).
10. When no calibrated BenchmarkScore exists yet for a task, the inferred tier (mined from field/harness data) fills the gap with a provisional signal; resolveEvidence always returns the calibrated verdict verbatim when one exists and only falls back to inferred data or hypotheses when it doesn’t.
11. The dashboard renders all of this as the mosaic matrix, leaderboard, evidence drill-down, and recommend explorer.

Tech stack

Next.js 16 (React 19, Tailwind 4) for the dashboard and API routes; BullMQ + Redis 7 for the run-execution queue and the atomic Lua-script cost gate; Prisma 7 over PostgreSQL 17 as the multi-tenant system of record. Model access is direct HTTP per provider — OpenAI and Groq via OpenAI-compatible chat completions, Anthropic via its native Messages API, and Ollama (on-device, keyless) via the same OpenAI-compatible path under a local provider — with no LLM SDK/framework in between. OpenTelemetry (gen_ai spans) instruments every model and judge call. zod validates environment/config at boot. A pnpm workspace (apps/web, apps/worker, packages/{core,db,config}) keeps execution, grading, and recommend logic in one shared package so the worker and the web app never diverge. Docker Compose runs Postgres and Redis locally.

Lessons

The reasoning-model capture bug. Reasoning models (gpt-oss) return their answer in a separate field from the standard completion field. The harvester was reading only the standard field, so gpt-oss runs silently recorded blank outputs — a run could finish, get graded, and land on the leaderboard while carrying no actual answer. Nothing errored; the pipeline just quietly scored emptiness. The failure mode wasn’t a wrong grade, it was a confidently-presented grade for output that was never captured.

The durable fix: always sanity-check output length against tokens spent before trusting a score. A completion that consumed real tokens but yields near-zero captured text is a capture-path bug, not a legitimate result, and should fail loud rather than feed the leaderboard. The general lesson carries beyond this one model family — any new provider or model shape that changes where the answer lives in the response is a fresh chance for the same class of silent-blank failure, so the check belongs at the capture boundary, not per-model.

The result is a benchmarking tool that’s honest about its own uncertainty — which, for a tool whose entire job is producing trustworthy numbers, is the only version worth building.