# Market Research Platform
Local-first research workspace for ingesting source files, tracing evidence, and drafting reports without sending private material to the cloud by default.
Status: active
## Problem
A research project mixes confidential internal strategy PDFs with public job-board exports and competitor decks — the kind of project shown in the platform's own "AI Market Research" workspace, where a single ingest zone holds both. The analyst needs to answer questions like "what have we already gathered about this competitor's hiring plans" without re-reading forty documents by hand, and the strategy material can't leave the machine to get an answer. General chat tools lose track of which claim came from which source; cloud research tools require uploading the documents to someone else's servers. For competitive and strategic material, neither is acceptable.
## Approach
The design bet is that ingestion should be a deterministic, zero-LLM pipeline — parse, chunk, embed, extract entities — so the same document produces the same chunks and the same entities on every run, and LLM calls are reserved for the analysis step, where non-determinism is the point. Everything downstream of ingestion (search, entity lookup, citations) is then reproducible and auditable independent of any model.
What was deliberately not done: no managed vector or graph database. The platform runs SQLite with the `sqlite-vec` extension (ANN search) plus FTS5 (BM25 keyword search) in one file, and an embedded Kuzu graph in another — both local, both single-process, neither requiring a network hop. It also dropped LangGraph as a pipeline dependency: ingestion, research, and reporting are all linear node sequences with exactly one conditional-edge shape (abort to end on error), so a ~100-line from-scratch async runner (`LinearGraph`) replaced the framework. And it does not use an LLM to extract entities or relationships during ingestion — spaCy's bundled statistical NER model runs offline and deterministically, at the cost of being limited to general-purpose entity types rather than domain-specific ones an LLM extractor could name.
## Architecture
> Uses the standard knowledge-graph pipeline (see [/toolkit/patterns/knowledge-graphs](/toolkit/patterns/knowledge-graphs)) — what differs here: entity extraction is deterministic spaCy NER over each ingested chunk, not LLM- or rule-based relation extraction, and the graph schema defines an entity-to-entity `RELATED_TO` edge that the current ingestion path never populates — graph queries traverse `Document -[MENTIONS]-> Entity` co-occurrence, not entity-to-entity relationships.
A multi-service Docker stack, local-first by design:
- **Frontend** — React + Vite + TypeScript (sources rail, search panel, report panel, settings)
- **Backend** — FastAPI + Python, orchestrating three pipelines (`ingestion`, `research`, `reporting`) through the in-house `LinearGraph` runner
- **Storage** — SQLite + `sqlite-vec` for vectors and FTS5 for keyword search (`vectors.db`), an embedded Kuzu graph for entities (`Entity`, `Document`, `MENTIONS`, `RELATED_TO`), and a separate SQLite file for project/source metadata
- **Document parsing** — in-process Python (PyMuPDF, python-docx/pptx, openpyxl, pytesseract OCR); docling is a PDF fallback but ships as an opt-in extra, not wired into the default parser routing (kept out of the slim install because it pulls in torch). A Node sidecar (`services/parser-sidecar`) handles only URL scraping now that file parsing moved in-process.
- **Models** — see below; two independently-tiered local generation paths (Ollama default, MLX opt-in) plus a cloud fallback
**Models — which, where, why:**
| Stage | Model / tier | Where it runs | Why there |
|---|---|---|---|
| Embeddings | Ollama `bge-m3` / `mxbai-embed-large` / `nomic-embed-text` (auto-picks the strongest installed, in that order) | `embed_node`, ingestion | Stronger semantic similarity on noun-heavy domain text than the bundled MiniLM, which collapses toward keyword overlap |
| Embeddings fallback | `all-MiniLM-L6-v2` (384-dim, bundled `sentence-transformers`) | `embed_node`, when Ollama is unreachable | Works fully offline, no daemon dependency — keeps ingestion functional with no local model pulled |
| Image vision description | Ollama multimodal tag if installed (`llama3.2-vision`, `llava`, `bakllava`, `moondream`, `gemma3`) | `parse_image`, ingestion (image uploads only) | Best-effort — the one ingestion path that is *not* zero-LLM; skipped silently if no vision model is pulled, so OCR alone still produces a usable chunk |
| Research analysis | Whatever `llm_factory.get_provider()` resolves (Ollama primary, MLX if `local_llm_provider=mlx`, Claude if `processing_mode=cloud_allowed` and Ollama is down) | `analyze_node`, research | Single call over already-retrieved context — no map-reduce needed, so it uses the general local/cloud provider rather than a dedicated tier |
| Report map step | Smallest installed match from `["qwen3:8b","llama3.1:8b","qwen2.5:7b","llama3.2:3b","mistral:7b"]` | `extract_brief` map phase, reporting | Runs once per ~9,000-char packed unit (adjacent chunks from one source), up to 4 concurrent calls — a fast small model keeps per-chunk latency down across the 20-40 units a typical transcript produces |
| Report reduce step | Strongest installed match from `["mistral-small:22b","gpt-oss:20b","gemma4:26b","qwen2.5:14b","phi-4:14b","gemma2:27b","qwen2.5:32b"]` | `extract_brief` reduce phase, reporting | One synthesis call over the clustered map output — worth the larger/slower model since it runs once, not per-chunk |
| Cloud fallback | Claude Sonnet | Any stage, only when `processing_mode="cloud_allowed"` and the local provider is unreachable | Opt-in only — the default `processing_mode` is `"local"`, so confidential documents never leave the machine unless the analyst explicitly allows cloud |
Local-model runtime: Ollama is the default backend (`local_llm_provider="ollama"`), reached over HTTP at `localhost:11434`, running on the host rather than in a container so it can use host GPU/Metal — the Docker Compose backend reaches it via `host.docker.internal`. MLX is an additive, opt-in alternative (`local_llm_provider="mlx"` / `embedding_provider="mlx"`), Apple-Silicon only, talking to an OpenAI-compatible local server (`mlx-serve` or `mlx_lm.server`, default `localhost:8080`) — default generation model `mlx-community/Llama-3.2-3B-Instruct-4bit`, default embedding model `mlx-community/bge-small-en-v1.5`.
Two different "default generation model" values exist in the repo and both are real, at different layers: `Settings.ollama_model` falls back to `mistral:latest` if unset, but the shipped `docker-compose.yml` overrides it to `qwen2.5:14b-instruct-q4_K_M`. Neither setting drives report generation, which resolves its own map/reduce tiers independently (table above) — the single `ollama_model` setting only matters for `analyze_node`'s one-shot research analysis call.
```mermaid
flowchart LR
subgraph Ingest["ingestion pipeline — deterministic, one LLM exception"]
P[parse_file] --> C[chunk_node]
C --> E[embed_node
Ollama/MLX embedder]
E --> N[ner_node
spaCy en_core_web_sm]
P -.image upload only.-> V[vision describe
Ollama multimodal, best-effort]
end
N --> VEC[(vectors.db
sqlite-vec + FTS5)]
N --> KG[(Kuzu graph
Entity / Document / MENTIONS)]
subgraph Research["research agent"]
RS[retrieve_semantic] --> AN[analyze_node
1 LLM call]
RG[retrieve_graph] --> AN
end
VEC --> RS
KG --> RG
AN --> OUT[analysis + citations]
subgraph Report["reporting pipeline — map-reduce"]
GF[gather_findings] --> MP["map: small model x N chunks
(4-wide concurrency)"]
MP --> RD["reduce: large model x 1
(top-12 chunks re-injected)"]
RD --> GO["generate_outputs
parallel, no LLM"]
end
VEC -.project chunks.-> GF
GO --> FMT["pptx / docx / email / notes / outline"]
AN -.cloud fallback.-> CL[Claude
cloud_allowed only]
RD -.cloud fallback.-> CL
```
This is a workspace a human operates — distinct from Stratagem, which is an autonomous research *agent*. The platform keeps the analyst in the loop: upload, search, review, and decide, with the documents never leaving the machine unless the analyst opts into cloud processing.
## Project lineage
Market Research Platform is the canonical continuation of the earlier **Market Research** app (`market_research`). The earlier Vite application proved the multi-source ingestion and local-search direction, but it exposed more model and storage choices in the client and carried an unresolved API-key boundary. This platform replaces it with a FastAPI service, deterministic ingestion, local SQLite and Kuzu stores, explicit provider policy, and source-level citations. The predecessor remains a private historical repository; it is deprecated rather than counted as a second active project.
## How it works
1. A source is queued and `parse_file()` routes by extension to an in-process parser (PyMuPDF for PDF, python-docx/pptx, openpyxl for Excel); URLs are the one case still sent to the Node sidecar's `/scrape` endpoint. A standalone image upload also runs OCR (pytesseract) and — the one non-deterministic step in ingestion — a best-effort vision description from an installed Ollama multimodal model, skipped silently if none is pulled.
2. `chunk_node` splits the parsed text with a from-scratch recursive splitter (`split_text`, `chunk_size=3000`, `chunk_overlap=200`, separators `["\n\n", "\n", ". ", " ", ""]`), page-aware when the parser returned multiple pages.
3. `embed_node` embeds each chunk (`embed_texts`) with the active embedder — Ollama (`bge-m3` / `mxbai-embed-large` / `nomic-embed-text`) by default, MLX if configured, falling back to the bundled `all-MiniLM-L6-v2` (384-dim) if neither is reachable — into a `(project_id, embedder_name)`-namespaced table in `vectors.db`.
4. `ner_node` runs `spacy.load("en_core_web_sm")` over each chunk (first 10,000 characters only, for latency), keeps entities labeled `ORG`, `PERSON`, `GPE`, `PRODUCT`, `EVENT`, or `WORK_OF_ART`, and drops names under 2 characters or already seen in that document (`seen_entities` — deduped per-document, not per-project).
5. Each surviving entity gets a stable id, `md5(f"{name}:{label}:{project_id}")`, and is written to Kuzu via `graph_store.add_entity()` (an `Entity` node) and `graph_store.add_mention()` (a `Document -[:MENTIONS {chunk_index, context}]-> Entity` edge, where `context` is the entity's containing sentence truncated to 200 characters).
6. At query time, the research agent's `retrieve_graph` node calls `graph_store.search_entities(query, project_id)` — a `CONTAINS` substring match against entity names, not embedding-based entity linking — and for the top 5 matches calls `get_related_entities()` (walks `RELATED_TO`, currently always empty) and `get_entity_documents()` (walks `MENTIONS` back to source documents).
7. `retrieve_semantic` independently pulls the top 10 vector hits from `vectors.db`; `analyze_node` merges both result sets into an analysis prompt (`general` / `swot` / `competitive` / `trend`), resolves a single provider via `llm_factory.get_provider()`, and sets `confidence = min(len(semantic_results) / 10, 1.0)` — confidence is scored from the semantic leg only, not the graph leg.
8. Citations returned to the UI carry `source_id`, `chunk_index`, and a 200-character snippet, so an answer traces back to the exact chunk, not just the source file.
9. Report generation (`gather_findings`) pulls every stored chunk and source record for the project — not scoped to one query — as input to a `StrategyBrief` extraction.
10. `extract_brief` resolves two independent model tiers unless the caller pins one: a map model (smallest installed match from a preference list favoring 3B-8B tags) and a reduce model (largest installed match from a preference list favoring 14B-32B tags) — see the Models table above.
11. The map step packs adjacent same-source chunks into ~9,000-character units and runs one call per unit against the map model, up to 4 concurrent (`DEFAULT_MAP_PARALLELISM`), extracting theme candidates, claims, quotes, and financials as JSON.
12. The reduce step clusters map output into top themes, re-injects the verbatim text of the 12 most-cited chunks (600 characters each) so the final synthesis call is grounded against real spans rather than a pure summary-of-summaries, and produces one `StrategyBrief` JSON via the reduce model.
13. `generate_outputs` fans out in parallel (`asyncio.gather`, no further LLM calls) to whichever formats were requested — pptx, docx, email (html), notes (md), outline (md) — each a pure-Python generator writing to `data/reports//`.
14. `processing_mode` gates cloud use end to end: `"local"` (the default) restricts every stage to the local provider; `"cloud_allowed"` tries local first and falls back to Claude only if the local provider is unreachable — confidential material only reaches Claude on an explicit, per-request opt-in.
## Tech stack
React, Vite, TypeScript on the frontend; FastAPI on the backend. `LinearGraph` — an in-house ~100-line async runner — replaces LangGraph for the three linear pipelines. Storage is SQLite throughout: `sqlite-vec` for ANN search and FTS5 for BM25 keyword search in one file (replacing an earlier LanceDB/pyarrow store), plus an embedded Kuzu graph for entities. spaCy (`en_core_web_sm`) does NER. Ollama is the default local model backend for both embeddings and generation, reached over HTTP on the host (not containerized, so it keeps GPU/Metal access); MLX is an additive, opt-in on-device backend for Apple Silicon, speaking an OpenAI-compatible API. Report generation runs its own map/reduce model-tiering independent of the single-call research path. Anthropic Claude Sonnet is the cloud fallback, used only when `processing_mode="cloud_allowed"` and local is unreachable. Docker Compose ties the services (backend, frontend, parser sidecar) together for local-first operation.
## Results
⚠️ no benchmark yet — there is no labeled query set or precision/recall comparison of graph-assisted retrieval against semantic-only retrieval.
Worked example, traced through the real pipeline: a source named `q3-competitor-brief.pdf` contains the sentence "Salesforce announced a partnership with Slack to expand its GPT-integrated workflows." `ner_node` extracts two `ORG` entities, `Salesforce` and `Slack`, each hashed into a stable id and written as `Entity` nodes with `MENTIONS` edges back to the document — `context` holds that sentence, `chunk_index` points at the exact chunk. A later query, "what is Salesforce doing with Slack," reaches `retrieve_graph`: `search_entities("Salesforce")` matches by substring, `get_entity_documents("Salesforce")` returns `q3-competitor-brief.pdf` with that same chunk_index and context, and it's merged into `analyze_node`'s prompt alongside the top semantic hits. But `get_related_entities("Salesforce")` returns an empty list — no `RELATED_TO` edge was ever written between Salesforce and Slack, because nothing in the ingestion path populates that table. The graph reliably answers "which documents mention X"; it does not yet answer "how are X and Y connected," despite the schema being built for it.
## Lessons
- Deterministic NER was chosen over LLM-based entity/relation extraction for ingestion because ingestion needed to be reproducible and free of LLM cost and latency — the same document yields the same entities on every run. The tradeoff is coarser entity types (spaCy's six statistical labels) and no relation extraction out of the box, which is why `RELATED_TO` exists in schema but sits empty.
- A from-scratch `LinearGraph` runner was chosen over keeping LangGraph as a dependency once all three pipelines turned out to be linear node sequences sharing one conditional-edge shape (abort-on-error) — carrying a general-purpose orchestration framework for that shape was more surface area than it bought.
- SQLite + `sqlite-vec` was chosen over LanceDB to collapse vector search and keyword search (FTS5/BM25) into a single dependency-light file, trading LanceDB's native vector-format tooling for one store that also namespaces by `(project_id, embedder_name)` so swapping embedding models never corrupts existing tables.
## Links
- [HTML project page](https://rosslabs.ai/projects/market-research-platform/)