RossLabs Agent Harness

In development. Pre-release, private repo, committed to daily (most recent commit 2026-07-18). The routing policy, gate profiles, and command surface below reflect the current state of main, not a finished v1 — interfaces and defaults are still moving.

Problem

A model call is only as trustworthy as the policy wrapped around it. Left on its own, nothing stops a cheap or open local model from returning garbage, nothing decides whether a task should hit the harness at all versus a deterministic solve, and nothing stops generated code from executing unsandboxed. Any of those left to the model’s own judgment reintroduces the exact unreliability the harness is meant to remove.

Approach

A portable, CLI-first agent harness that treats the model as a swappable data row — local (Ollama / MLX / llama.cpp) or cloud (OpenAI, Groq, Fireworks, OpenRouter, Together, DeepInfra). The value isn’t the model; it’s the deterministic scaffolding around it, encoded in Rust so the model never computes policy.

Architecture

Part of the verification-gated agent-work pattern (see /toolkit/patterns/verification-gated-agents) — here: a per-task oracle drives two decisions at once — whether to invoke the full agentic tool loop at all vs. a deterministic or Program-Aided solve, and (when a model does run) which model, escalating away from cheap/local only when measured evidence says it must.

A single-binary Rust workspace (10 crates under one Cargo workspace: provider, tools, gates, context, engine, vault, cli, eval, bench, llmwiki) built around one event-sourced step() loop — rehydrate session → assemble prompt → one model call → parse tool intents → gate → execute → append observation → repeat. The model never computes policy; every decision that determines whether and which model runs, what it’s allowed to touch, and whether its output is accepted is a deterministic Rust function, not a prompt.

flowchart LR
  A["harness run &lt;task&gt;"] --> B["Task planner<br/>derive_solution_mode()"]
  B -->|deterministic transform| D["direct<br/>no model call"]
  B -->|compute task + verify_cmd| E["code_exec<br/>Program-Aided: model writes + runs code"]
  B -->|reliability cell: 6+ attempts, reliable| F["one_shot<br/>single model call"]
  B -->|coding/write or unproven| G["agentic<br/>full tool loop"]
  G --> H["provider::routing::route()<br/>role/task to model"]
  H -->|policy override| P1["pinned model"]
  H -->|reliable / above pass-rate floor| I["Local: Ollama / MLX / llama.cpp"]
  H -->|unreliable or below pass-rate floor| J["Escalate: bigger local, then cloud<br/>OpenRouter / OpenAI / Groq / Anthropic"]
  I --> K["Tool-call ladder<br/>native to schema-constrained to prompt+repair"]
  J --> K
  K --> L["Gate pipeline (deny-first)<br/>SAFE / RISKY / DECISION / PRODUCTION"]
  L -->|deny / unresolved ask| M["Blocked, fails closed"]
  L -->|allow| N["Tool exec: read/write/edit/bash/grep/test"]
  N -->|--sandbox| O["Docker to Seatbelt(macOS) to Bubblewrap(Linux) to refuse"]
  N --> Q["Session tree (JSONL)<br/>resume / fork / tree"]
  E --> R["Verification gate<br/>structural + source-fidelity"]
  N --> R
  R -->|reject| H
  R -->|pass| Q
  Q --> S["Compaction<br/>section-schema summarize, gate log kept verbatim"]
  T["harness bench"] -.folds pass/fail into.-> U[("reliability-*.json<br/>per model x task_class")]
  U -.read by.-> H

Models — which, where, why (rows drawn from .harness/models.json’s live routes table, not an exhaustive list):

Route	Model	Where it runs	Why there
`default`	`qwen3:8b-q4_K_M`	Ollama, local	Small, always-pulled smoke baseline — kept stable so first-run UX and `scripts/e2e_smoke.sh` never depend on a large model download
`coding/read`, `coding/write`	`qwen3-coder:30b`	Ollama, local	Top-evidenced local coder among pulled models per live capability probe (`harness models use`), verified 2026-07-03
`cheap`	`google/gemini-2.5-flash-lite`	OpenRouter, cloud	Cost floor for tasks that don’t need a strong model
`code-cloud`	`moonshotai/kimi-k2.7-code`	OpenRouter, cloud	Cloud coding fallback when the local coder’s reliability cell is unreliable or a task needs more context than fits locally
`experiment`	`nvidia/nemotron-3-super-120b-a12b`	OpenRouter, cloud	The only OSS model that passed the vault-ingest fidelity gate in the empirical bench (85% fidelity) — cheapest of the reasoning models tried, not the premium-priced ones
`think`	`claude-sonnet-5`	Anthropic, native (`/v1/messages`)	Top empirical performer on the same vault-ingest bench (95% fidelity)
`deepthink`	`claude-opus-4-8`	Anthropic, native	Escalation ceiling when local and cheap-cloud both fail their oracle
`vision`	`qwen/qwen3-vl-235b-a22b-instruct`	OpenRouter, cloud	Multimodal route for image-bearing tasks; no local vision model is in the default registry

Local vs. cloud is a provider-trait seam, not a hardcoded branch: provider::cloud gives every OpenAI-compatible host (OpenAI, Groq, Fireworks, OpenRouter, Together, DeepInfra) one shared request/response mapping keyed by (provider name, base_url, api-key env var); Anthropic gets its own native mapping since its wire format isn’t OpenAI-compatible. route()’s escalation only fires when a .harness/routing.json policy opts in — the default policy (privacy: allow, empty escalation rule) means a fresh install can never send code off-device until a human explicitly configures it, and privacy: strict refuses cloud outright regardless of any other setting.

Tools & infra — which, why:

Rust workspace, single static binary (harness) — no interpreter tax, no daemon IPC; crates/cli is the only crate with a main.rs, everything else is a library the CLI composes.
gates — a deny-first permission classifier (SAFE/RISKY/DECISION/PRODUCTION tiers) ported from build-loop’s Python classify_action.py/autonomy_gate.py into ~1.7K LOC of Rust (excluding tests), evaluated in-process before any tool executes, not left to the model to self-police.
tools — the typed primitive set (read/write/edit/bash/grep/test) plus the sandbox dispatch: --sandbox routes bash/test through exec::run_capture, trying Docker (docker info succeeds + image present — the only backend with --memory/--cpus/--pids-limit caps), falling back to macOS sandbox-exec (Seatbelt) or Linux bwrap (Bubblewrap) — fs/network isolation, no resource caps — and refusing to run at all rather than executing unsandboxed, unless --allow-no-sandbox is passed explicitly with a printed warning.
context + NavGator — reads .navgator/architecture/ into an in-memory context graph for repo-map assembly; a missing or malformed map degrades to “unavailable” rather than panicking, since context absence must never kill the loop that calls it.
bench — the oracle-measurement loop: re-invokes harness run as a real subprocess per fixture task (so a hung run is genuinely killable, not just in-process), runs the fixture’s own verify.sh, and folds results into per-(model × task_class) reliability cells (reliable/usable/unreliable/insufficient_data + pass rate) that routing::route() reads to decide when to escalate.
vault — the concrete instance of the gated-generation pattern: a structural gate (verify.rs — required frontmatter keys, allowed type/status values) plus a source-fidelity gate (fidelity.rs — at least 20% of the source document’s top 20 salient terms must appear in the output), so a well-formed page about the wrong subject still gets rejected.
llmwiki — the one crate wired to OpenTelemetry/OTLP for direct trace export; the rest of the engine doesn’t yet emit spans (tracked, not shipped).
tiny_http — backs harness serve, a local dashboard for routing/model state and ad hoc queries; no external web framework.
Rally Point (agent-rally-point, the rally CLI) — not a harness dependency (absent from every crate’s Cargo.toml); it’s the multi-agent coordination tool (claims, handoffs, presence) used to develop the harness and the design target for a future bidirectional multi-LLM-team mode, reused rather than rebuilt.

How it works

harness run "<task>" loads .harness/models.json (capability registry — falls back to a built-in seed if absent or malformed) and .harness/routing.json (routing policy — an absent file means local-only, never escalates).
derive_solution_mode() classifies the task before any model runs: a known deterministic transform short-circuits to direct (no model call at all); a compute task with a checkable verify_cmd routes to code_exec (the Program-Aided lane — the model writes and executes code, graded against an expected answer within a tolerance, up to 2 attempts); a role/task whose reliability cell has 6+ recorded attempts and a reliable verdict gets one_shot (skip the full loop, one call suffices); a workspace-mutating coding/write task, or anything with no cheaper mode proven, falls to agentic — the full tool loop.
For one_shot/agentic, provider::routing::route() picks the model: an explicit .harness/routing.json override wins outright; otherwise it reads the folded reliability cell for (local model, role_task) from .harness/bench/reliability-*.json (produced by step 9 below) — an unreliable/insufficient-data verdict or a pass rate below the policy’s threshold escalates to a bigger local model first, then to the best-fit cloud model, in that order. privacy: strict bypasses all of this and refuses cloud unconditionally.
The chosen model is called through the provider trait — local via Ollama’s /v1-compatible endpoint at localhost:11434 (Metal-accelerated on Apple Silicon) or MLX/llama.cpp, cloud via the shared OpenAI-compatible client (OpenRouter/OpenAI/Groq/Fireworks/Together/DeepInfra) or Anthropic’s native endpoint — through a tool-calling ladder that degrades native function-calling → schema-constrained (GBNF) → prompt-parsed-with-repair for weaker models or smaller quants.
The engine’s step() loop parses the model’s tool intents and evaluates each one through the gates pipeline before execution: a deny-first classifier resolves SAFE/RISKY/DECISION/PRODUCTION to allow/ask/deny; deny and any unresolved ask fail closed.
An allowed call executes through the typed primitive set (read/write/edit/bash/grep/test). With --sandbox, bash/test run inside the first available isolation backend — Docker, then Seatbelt (macOS), then Bubblewrap (Linux) — never silently on the bare host.
The observation is appended to the session tree (one JSONL file, id/parentId per entry — the structure resume/fork/tree navigate); when context exceeds the token budget, compaction folds old turns into fixed sections (Goal/Constraints/Progress/Decisions/Next/Critical), with every gate-decision entry kept verbatim, never summarized away.
Where a task has a checkable oracle — vault ingestion is the concrete, measured case — the produced output passes through a two-part gate before it’s accepted as done: a structural check (required frontmatter keys, allowed field values) and a source-fidelity check (does the output share enough of the source’s salient terms). On an OSS ingest sweep this gate rejected 6 of 7 garbage/wrong-subject outputs a raw model call would otherwise have returned as a “successful” page.
harness bench closes the loop offline: it replays fixture tasks through harness run as subprocesses, scores them against each fixture’s own oracle, and folds the results into the reliability cells that step 3 reads — so routing escalation is grounded in measured pass rates per (model, task_class), not a static price tier.
harness serve exposes the same routing and model state through a local dashboard for inspecting live decisions and running ad hoc queries.

Tech stack

Rust, single-binary CLI (harness) over a 10-crate Cargo workspace — provider (capability registry, routing, tool-call ladder), gates (deny-first classifier, ported from build-loop), tools (typed primitives, sandbox dispatch), context (NavGator-backed repo map), engine (event-sourced loop, session tree, compaction), vault (gated document ingestion), cli, eval, bench (fixture runner, reliability folding), llmwiki (OTel/OTLP export). Model backends: local Ollama / MLX / llama.cpp; cloud OpenAI-compatible (OpenRouter, OpenAI, Groq, Fireworks, Together, DeepInfra) plus Anthropic Claude native. Sandboxing: Docker (default, resource caps) with macOS Seatbelt / Linux Bubblewrap fallback. tiny_http for the local dashboard. NavGator for the repo-map graph; Rally Point for multi-agent build coordination.

Results

✅ 6 of 7 — on an OSS ingest sweep against the vault’s structural + source-fidelity gate, 6 of 7 open-source models’ outputs were rejected as garbage or wrong-subject (hallucinated subjects, empty runs, generic content); not one garbage page passed silently.

✅ In the same bench, only 1 of 7 OSS models passed outright — nvidia/nemotron-3-super-120b-a12b at 85% fidelity, the cheapest of the reasoning models tried, not the premium-priced ones (glm-5.2, deepseek-v4-pro, minimax-m3 all failed to even emit a page). claude-sonnet-5 topped the same bench at 95% fidelity.

⚠️ No benchmark yet on aggregate routing cost savings, sandbox-fallback frequency in practice, or the accuracy of the direct/code_exec/one_shot/agentic mode split across a broader task set — only the vault-ingest gate-rejection and per-model fidelity numbers above have been measured.

Lessons

Self-hosting is the design bet: once the core loop is reliable on a local model, the harness builds its own later phases, rather than treating self-hosting as a stretch goal to revisit after the harness is “done.”
Price ≠ task fitness — benchmark, don’t assume. The vault-ingest sweep’s cheapest reasoning model outperformed three higher-priced “thinking” models that failed to produce output at all; harness bench and reliability-cell-gated escalation exist because a static price-tier router would have picked the losers.
Fail closed on the axis you can’t fully control. Both --sandbox and cloud escalation share the same shape: an absent capability (no Docker, no explicit routing policy) resolves to the safer default (refuse-with-opt-in, local-only) rather than silently degrading to the riskier one.