# Spectra

Captures a running app on web, macOS, iOS, or watchOS, then turns the raw screenshots and clips into a tagged, exportable demo — no editor required.

Status: active

## Download

Native macOS builds are published in the public [Spectra releases repository](https://github.com/tyroneross/spectra-releases/releases). The plugin source remains available from the repository linked above.

## Install

```bash
claude plugin marketplace add tyroneross/RossLabs-AI-Toolkit
claude plugin install spectra@rosslabs-ai-toolkit
```

## The Problem

Marketing captures span platforms that each demand their own stack: Chrome DevTools for web, Accessibility for macOS, `simctl` for iOS, and simulator workflows for watchOS. Stitching together screenshots, short clips, usage sequences, and exports across all of them is manual work that does not scale.

## What I Built

Spectra is the primary Ross Labs capture tool for product demos: screenshots, guided usage sequences, smart framing, and a tagged export library across web, macOS, iOS, and watchOS. Video capture is supported where available and expanding.

## Architecture

Spectra is an MCP server plus a local daemon — it exposes UI automation and capture as tools a host agent calls, rather than running its own planning loop. The primary path is host-routed: Claude Code, Codex, or another MCP-speaking coding agent reads a snapshot, decides the next action, and calls Spectra's tools to execute it. Spectra.app, a standalone macOS menu-bar fallback, only plans locally (via Claude Haiku) when no host agent is present.

```mermaid
flowchart LR
  H["Host agent (Claude Code / Codex)<br/>plans from spectra_snapshot"] -->|MCP tool calls| D
  SA["Spectra.app standalone<br/>WalkthroughPlanner + Claude Haiku 4.5"] -.fallback, no host agent.-> D
  D["spectra_connect<br/>driver select"] --> CDP["CdpDriver (web)"]
  D --> NAT["NativeDriver (macOS AX)"]
  D --> SIM["SimDriver (iOS / watchOS)"]
  CDP --> SNAP["spectra_snapshot<br/>AX / DOM tree"]
  NAT --> SNAP
  SIM --> SNAP
  SNAP -.AX empty or thin.-> VIS["Vision fallback<br/>Apple Vision framework, on-device"]
  VIS --> STEP
  SNAP --> STEP["spectra_step<br/>Jaro-Winkler resolve engine"]
  STEP --> ACT["spectra_act<br/>driver executes"]
  SNAP --> AN["spectra_analyze<br/>6-factor importance scoring"]
  AN --> CAP["spectra_capture<br/>clean + auto-frame"]
  ACT --> CAP
  CAP --> LIB[("Library<br/>.spectra/library/index.json + media/")]
  CAP --> DEMO["spectra_demo<br/>ffmpeg spotlight + captions"]
  DEMO --> LIB
```

**Models — which, where, why:**

| Stage | Model / tier | Where it runs | Why there |
|---|---|---|---|
| Host-routed planning (primary path) | None — the host LLM (Claude Code, Codex, or another MCP-speaking coding agent) plans | Outside Spectra; the host reads `spectra_snapshot` output and calls Spectra's MCP tools | Keeps Spectra a pure automation/capture layer — no API key, no model cost, no planning logic duplicated inside the tool for the path almost all sessions use |
| Standalone fallback planning | Claude Haiku 4.5 (default, overridable per call) | `WalkthroughPlanner.swift` + `AnthropicClient.swift`, `Spectra.app` only, one non-streaming call per turn | Small, fast model for single-turn action-plan JSON when no host agent is driving; temperature 0 for deterministic plans; the user's own API key lives in Keychain and never reaches the daemon |
| Element-grounding fallback | Apple Vision framework (on-device, no network call) | `computer-use/vision-fallback.ts`, gated on AX-node-count (fires only when the AX tree is empty or thin) | AX reads take ~30–80ms and come pre-labeled; screenshot-grounding is ~5–50x slower and, per the code's own cited benchmark, under 70% accurate on ScreenSpot-Pro — so vision is a fallback, never the default |
| Natural-language element resolution | Jaro-Winkler string similarity — deterministic, no model | `core/resolve.ts`, backs `spectra_step` | Matching "click the Log In button" to an AX label is a string-similarity problem, not a reasoning problem; resolving it without a model call keeps it instant and free |
| Screen importance scoring | 6-factor weighted heuristic (fixed weights, derived from UEyes CHI 2023 eye-tracking research) | `spectra_analyze` | Role, position, interactivity, label quality, density, and prominence combine into a fixed formula — reproducible scoring with no inference cost |
| Change / scene detection | dHash perceptual hash + AX structural diff — deterministic | `intelligence/change.ts`, `spectra_demo action=scan` | A binary "did the screen change" signal doesn't need a model; a gradient hash and a tree diff answer it directly |

**Models: minimal by design.** Spectra runs model-free for its default host-routed usage; the one LLM in the stack (Claude Haiku 4.5) only activates in the standalone-app fallback path, and even the vision-grounding fallback is a local Apple framework, not a hosted model.

**Tools & infra — which, why:**

- **MCP server** (`@modelcontextprotocol/sdk`) — exposes 15 tool handlers (`connect`, `snapshot`, `step`, `act`, `capture`, `demo`, `analyze`, `discover`, `session`, `library`, `llm-step`, `record`, `walkthrough`, `replay`, `computer-use`) over stdio to any MCP-speaking host.
- **Node.js daemon** (`node:http`, custom JSON envelope in `src/contract/wire.ts`) — a long-running process holding driver connections and sessions, reachable over a Unix domain socket by default or optional loopback TCP with bearer-token auth, so the CLI/MCP process and the daemon can run as separate PIDs.
- **`CdpDriver`** — a hand-rolled Chrome DevTools Protocol client (own connection handling plus 7 domain wrappers), not Puppeteer or Playwright — keeps the web driver dependency-light for a narrow automation surface.
- **`NativeDriver` / `SimDriver`** — a Swift binary (`spectra-native`, built via `swiftc` against Foundation, ApplicationServices, AppKit, CoreGraphics, Vision, ScreenCaptureKit, AVFoundation) bridging the macOS Accessibility API and `simctl` for iOS/watchOS.
- **FFmpeg** — external binary, resolved via `which ffmpeg` rather than bundled — handles transcode, spotlight-focus rendering, lower-third caption burn-in, and scene-change scanning inside `spectra_demo`.
- **Zero-dependency PNG codec** built on `node:zlib` — decode/encode/crop/resize without pulling in `sharp` or `jimp`.
- **Storage** — no database. A flat `.spectra/` directory per project (found by walking up to `.git`, `package.json`, or an existing `.spectra`, else `~/.spectra`) holds `sessions/<id>/` for ephemeral step recordings and `library/index.json` + `library/media/<capture-id>/` for the persistent tagged catalog; index writes are atomic (tmp file + rename).
- **ScreenCaptureKit + AVFoundation** (Swift, `spectra-composite-capture` binary) — native screen recording with hardware H.264/HEVC encoding.
- **Next.js dashboard** (`web-ui/`) — local-only browse/manage/export UI served at `:4300`, separate from the MCP/daemon runtime.
- **Vitest** — 30 test files, 329 tests per the repo's own README, run via `npm test`.

## How it works

1. A host agent (or `Spectra.app` standalone) calls `spectra_connect` with a target — a URL, an app name, or `sim:device` — which selects a driver: `CdpDriver` for web, `NativeDriver` for macOS, `SimDriver` for iOS/watchOS.
2. `spectra_snapshot` returns the current accessibility tree: a compact element list with roles, labels, bounds, and available actions. If the AX tree comes back empty or thin, the Apple Vision on-device fallback grounds the screen from pixels instead and returns AX-shaped nodes so the rest of the pipeline doesn't need to know which source produced them.
3. In the host-routed path, the host LLM reads that snapshot and decides the next action itself, then calls `spectra_act` or `spectra_step` directly — no model runs inside Spectra for this step. In the standalone-app path, `WalkthroughPlanner` sends the snapshot (plus an optional screenshot) to Claude Haiku 4.5 and parses the returned action plan.
4. `spectra_step` takes a natural-language intent ("click the Log In button") and resolves it to a specific element with the Jaro-Winkler string-similarity engine, returning a confidence score alongside the resolved element ID.
5. `spectra_act` executes the resolved action (click, type, clear, select, scroll, hover, focus) on the driver and returns the updated snapshot.
6. `spectra_analyze` scores every element on screen with the 6-factor importance heuristic and groups them into labeled regions (Navigation, Form, etc.), also classifying overall UI state (loading, error, empty, populated).
7. `spectra_capture` takes the screenshot or recording. Visual cleanup (hide scrollbars, clean the simulator status bar, strip cursor artifacts) runs first if requested, then framing — `full`, `element`, `region`, or `auto` — crops to the relevant content using the importance scores from step 6.
8. `spectra_discover` automates steps 2–7 across an entire app: BFS-crawls navigable elements, captures and frames each new screen, deduplicates by role+label fingerprint, and flags (without capturing) any screen showing password or credential fields.
9. `spectra_demo` turns a raw recording into a polished clip: `scan` runs dHash-based scene-change analysis to find active segments, `polish-clip`/`polish-script` apply spotlight focus and burn in numbered lower-third captions via FFmpeg, and `run-script` can drive a live CDP browser through scripted beats while capturing.
10. Finished captures land in `.spectra/sessions/<id>/` (ephemeral) or get promoted into `.spectra/library/` (persistent, tagged) via `spectra_library action=add` — a flat JSON index plus a `media/<capture-id>/` file, no database involved.

## Tech stack

Node.js + TypeScript for the MCP server, daemon, and drivers; Swift for the native macOS/iOS/watchOS bridge (Accessibility API, `simctl`, ScreenCaptureKit, Apple Vision framework). The web driver is a hand-rolled Chrome DevTools Protocol client, not Puppeteer/Playwright. Element resolution (Jaro-Winkler), importance scoring, and change detection are all deterministic algorithms — no model in the loop. The one LLM in the stack, Claude Haiku 4.5, runs only in the standalone `Spectra.app` fallback planner when no host agent is present. FFmpeg (external binary) handles video transcode and demo-clip production. Storage is a flat `.spectra/` directory — JSON index + media files, no database. A Next.js dashboard serves a local browse/export UI.

## Impact

Spectra supersedes Showcase. Showcase proved the value of development-time capture; Spectra makes that workflow cross-platform and organized enough to become the long-term demo library. The site should now treat Spectra as the default capture recommendation and Showcase as the legacy migration path.

## Worked Example

Consider a demo that needs to show the same product flow on its web app and its macOS companion. Previously this required Chrome DevTools scripting for the browser session and a separate manual walkthrough for the native app, with no shared place for the resulting assets.

With Spectra, a single capture session spans both surfaces. For the web leg, the Node runtime attaches over Chrome DevTools Protocol and steps through the flow, taking screenshots at each point in the guided usage sequence. For the macOS leg, the Swift native bridge drives the same kind of walkthrough through the Accessibility API, so the app does not need scripted UI hooks to be captured. iOS/watchOS legs of the same flow route through simulator control instead.

Each screenshot in the session carries framing metadata and tags, and the full set — web and macOS together — lands in one local library entry with an export manifest, rather than in two disconnected asset folders. This is the mechanism that lets Spectra replace Showcase as the default: Showcase's development-time capture covered one platform at a time; Spectra's session model is what makes the cross-platform library possible.

Capture duration, session count, and library size: ⚠️ no benchmark yet — the write-up does not report usage figures for this workflow.

## Links

- [HTML project page](https://rosslabs.ai/projects/spectra/)
- [Source code](https://github.com/tyroneross/spectra)