The problem and its constraints
A robot fleet streams policy decisions all day. When one robot does something strange, "what happened" is usually answered by scrubbing video and guessing. Ground Control's premise is that a logged decision should be a first-class, interrogable object: replay it with a different parameter, reroll it through real physics, search the fleet's history for decisions that are one perturbation away from failing, and gate policy updates on that evidence.
Three constraints shaped everything:
- Every claim must carry its evidence. A dashboard that shows a number without saying how it was obtained trains operators to distrust it. Anything the platform asserts is labeled by how it was produced, and the label can only be earned, never assumed.
- One host, deliberately. This is an operator workbench, not a fleet-scale streaming pipeline. One API process, worker processes beside it, one database file. The scale path (brokers, multi-machine ingest) exists as a written appendix in the design docs, not as half-built code.
- Physics is optional at runtime; honesty is not. The MuJoCo assets that power physics rerolls are heavy and stay out of the repo. A fresh clone must still run everything else, self-skip what it can't prove, and say plainly which evidence tier it's operating at.
System shape
SOURCES robot policy decisions ROS2/JSONL · MCAP · live capture daemon (60Hz+, batched) · guest WebSocket lane (bring-your-own-robot) │ ▼ INGEST schema validation · dedupe · per-decision physics readiness │ ▼ STORE event store: SQLite (WAL) or Postgres behind one dialect seam decisions · jobs · fragility scans · incidents · promotions │ │ ▼ ▼ API FastAPI · REST + WORKERS separate processes: WebSocket (coalesced replay · MuJoCo physics ticks, cursor resume) reroll · policy analysis │ │ claim → execute → report ▼ ▼ FLIGHTDECK operator UI ARTIFACTS renders · comparisons 14 panels, guided tour · manifests on disk
One mental model: the database is the coordination layer. The API and every worker are separate OS processes that share only the store and the artifact tree. Workers claim jobs, execute, and report through the store's durable-job tables, with a watchdog that abandons stuck handlers. There is no message bus to operate, and any piece can be restarted independently.
The decisions that carry load
1. Evidence tiers are the type system of the product
Every result is tagged with one of three tiers: policy_only
(what the policy would have chosen), physics_reroll (the
counterfactual actually ran through closed-loop MuJoCo), or
physics_validated (the reroll also passed baseline-parity
checks). The vocabulary lives in a single module, a label-parity test keeps
the UI, reports, and narratives using identical wording, and nothing anywhere
may present a lower tier as a higher one. Fragility search is built on the
same idea: a cheap policy-only pre-rank chooses candidates, physics rerolls
confirm them, and results are ranked by the tier of proof they achieved.
2. The replay surface can't overpromise
Whether a decision can be replayed — and at which tier — is answered by one shared pre-flight resolver: capability checks, the badges on every decision list, and replay execution itself all ask the same code, so the UI can never offer a replay the engine would refuse. A parity test sweeps every replay parameter against every artifact state to keep that promise from drifting as either side evolves.
3. The promotion gate has no allegiance
The improvement flywheel closes the loop: fragility findings compile into evaluation suites, a candidate policy is retrained against them, and a promotion gate rerolls both baseline and candidate through physics before the candidate may ship. The system's own first retrained candidate was blocked by its own gate — the physics evidence didn't clear the bar, and the committed validation report says exactly that. Reporting that straight, rather than tuning the gate until it passed, is what makes the gate worth anything.
4. The cognition layer had to earn its switch
The platform estimates what each decision was worth via a distilled
Q-ensemble. The numbers are labeled q_proxy_return, never
"confidence"; every Q bundle is bound to the policy it was distilled from
(a mismatched policy gets a typed refusal, not a stale score); and the whole
layer sits behind a calibration verdict. It's enabled today because a
calibration gauntlet earned it: per-state ranking hit 0.607 top-1 accuracy
against a 0.20 random baseline, and an 18-of-20 physics cross-check agreed
(p = 0.0002). If a future verdict fails, the UI visibly drops to
distribution-only status with the reason shown — a kill switch that
explains itself.
5. Leave-anytime ingest
The bring-your-own-robot lane was designed backwards from the exit. Guests
mint scoped tokens (stored as hashes, capped robots, message and byte
budgets, 72-hour TTL) and stream over a WebSocket with per-batch
acknowledgements and honest drop counters. Revoke is never blocked by rate
budgets, severs live streams server-side, and can purge every row the guest
ever sent — with server-verified counts as the receipt. The client is a real
package — pip install robot-fleet-connect —
and it vendors the platform's mapping core byte-for-byte, with CI failing
if the two copies drift.
6. Docs with a tripwire
The internal architecture doc is canonical: a claims-to-evidence table maps every product claim to the code that implements it, the test that pins it, and the validation artifact that demonstrates it — and a doc-parity script fails CI when any pointer rots. The same discipline shapes the incident narratives the platform exports: deterministic templates over stored evidence, with refusal-by-construction for anything the evidence doesn't support. No LLM in the loop, so a narrative can be trusted verbatim.
Numbers that keep it honest
- 560+ tests, and a machine without the heavy physics assets still passes 420+ of them with zero failures — asset-dependent tests self-skip and say why.
- Two robot morphologies, one adapter contract: a Unitree G1 humanoid and a Go1 quadruped run the same capture → replay → reroll → promotion workflows.
- Dual-engine store: the full suite runs green on both SQLite and Postgres in CI, behind one dialect seam.
- Live spine: capture batching at 60Hz+, WebSocket fan-out as 250ms coalesced ticks with cursor-based resume after disconnects.
- Every claim on camera: scripted walkthrough videos regenerate the UI evidence after each phase, and the 3-minute public walkthrough shows the real system end to end.
What I'd do differently
The store's dialect seam came late. SQLite-first was right for a single-host tool, but the Postgres seam was retrofitted after the store had grown a dozen table families — building the seam earlier would have made that a non-event.
Vanilla JS is reaching its ceiling. The no-build-step SPA kept iteration instant across fourteen panels, but the panel-to-panel contracts now live in my head. The next structural investment is types.
Naming is archaeology. The core package still carries the name of the phase that created it, several phases ago. Harmless, but every new reader asks the same question, and renames get more expensive every week.
The first-clone wow moment needs assets. Keeping MuJoCo assets out of the repo is correct hygiene, but it means the most impressive tier is the one a fresh clone can't produce. Committed physics recordings — real MP4s replayed exact-match without inflating the evidence tier — recover most of it, and were designed for exactly this reason.