Control plane — the thin orchestrator

The principle: the genius is in the math, not in fat workers. The control plane moves data and runs cheap math. It never inspects content, never validates payloads, never keeps a fat worker in the write path. Reliability is emergent — liars get deprecated and trustworthy devices overwrite them — so the machinery that moves the data can be almost nothing.

Three boxes (and one sideways tap)

1. Orchestrator — tiny (256 MB / 0.25 vCPU, alwaysdata). It is three things and nothing else:

   • the tensor in RAM (a small sparse array — KB to low-MB),
   • a priority queue built from the DB — never done: it always hands out jobs, ranking targets by f(recency of the last result, trust of who observed it, provider-expected volatility — per-posting volatility aggregated over a provider —, …),
   • a cheap fold that turns incoming measurements into updated weights (moving average / median — arithmetic).

   It is DB-connected (over CrateDB's HTTP SQL port via curl, so the static-musl binary stays lean — no embedded TLS/pg stack). It serves devices directly: a device coming online makes one call and gets back both its copy of the current tensor and a lease (a work package). Contact is loose and non-permanent. Near-zero compute; maximum leverage — the tensor it hands out steers every device's behaviour.

2. Database (CrateDB) — the corpus. Exactly one row per URL: the last observation. No history, no per-URL vote tree. Plus a per-device trust score — a number that weights queue priority, never a gate. The write is last-writer-wins, no content gate. CrateDB has no row-level security, so a device can't be handed a CrateDB credential scoped to its one lease — instead it deposits to Corbet S3 via a presigned PUT the orchestrator mints (the URL is the lease write key — object-scoped, expiring, no credential in the device), and the orchestrator folds S3 → CrateDB out of band (async/batched; the only corpus writer). An independently scoped B2 URL is the overflow path when Corbet is unreachable. Deposits are plaintext (public job content) + thin.

3. Render — pure frontend: the webapp, the UI, the jobcache front-end. No control plane. (Today's jobcache-interface is a monolith that also does leasing + ingest; that part moves out — see Migration.)

Sideways tap: devices and the DB emit telemetry to Axiom; an ops dashboard reads it elsewhere. The orchestrator does not carry observability either.

The device loop

device online ──one call──▶ ORCHESTRATOR ──▶ { tensor, lease, primary + overflow grants }
                                 │  (reads DB for the queue, folds measurements,
                                 │   folds an object-store inbox → CrateDB)
                                 ▼
 Corbet deposit inbox ──COPY→UPSERT one row LWW→DELETE────────────▶ CrateDB
 B2 overflow inbox ─────same-key safety-net fold─────────────────▶ CrateDB
        ▲
        │ device PUTs its one observation to the presigned (object-scoped) URL
   device scrapes per the tensor's weights
   loosely: deposit, re-poll for new weights + next lease

A device talks only to the orchestrator (control) and an object-scoped storage URL. It never touches Render or CrateDB. The orchestrator serves the tensor itself rather than publishing it to B2/CDN because the device has to come to the orchestrator for its lease anyway — bundling the tensor (and the deposit URL) into that same on-online call is free. One call per device-online is cheap even at high fleet counts; if it ever strained the box, front the read with D1 (edge). No publish step.

Causal routing receipt on every participant outcome

Every modern lease carries one Sporewright RoutingReceipt: the exact tensor revision, addressed decision, ranked alternatives, selected execution class, dimension values/weights, and contribution-origin addresses that caused the offer. Browser, desktop, and container participants echo that same immutable receipt on all outcome paths:

preflight/local veto -> POST /outcome + receipt -> release only
executed failure     -> POST /outcome + receipt -> environment → … → corpus update
discovery result     -> POST /discovered + receipt -> durable outcome + field update
observation deposit  -> S3 deposit includes receipt -> CrateDB fold + field update

A local preflight veto cannot masquerade as class feedback. The orchestrator validates its selected option and target address, releases the lease, returns an explicit empty projection path with responsibility=participant-local, and writes no routing event. Missing or mismatched adapter revisions are preflight vetoes; a container never silently runs different adapter code against the offered binding.

Executed outcomes retain the full receipt in routing_outcomes.routing_receipt. The flattened operation/source/binding/task/environment/revision columns remain the bounded aggregate and recovery index. The response distinguishes the decision revision from the new tensor revision and explicitly reports the leaf-to-root responsibility path. This lets operators prove which decision an outcome trained without making the hot scheduler replay the event log.

The receipt revision is an opaque causal identity, not merely the reconstructed in-memory outcome count. One durable CrateDB counter increment at orchestrator boot namespaces that process generation; all decisions inside it then use the local field revision. Restarts therefore cannot reuse one label for two tensor states, while routing still performs no revision write on its hot path. If the counter is unavailable, a collision-resistant ephemeral generation is reported explicitly instead of silently reusing the legacy namespace.

Class curiosity is also paired with a concrete task before enqueue. JobCache compares the bounded candidate window using each class alternative's uncertainty-adjusted tensor score, reserves the most informative viable (task address, class) pair, and admits that task into the fixed-size publish batch even when it sits below the normal priority cutoff. A known-hostile first URL can therefore no longer consume every browser probe while an uncertain, lower-risk URL waits below it.

Reliability — the priority queue, not a vote and not a cut-off

One value per URL, no history. Reliability is the orchestrator's priority queue:

• No device is ever cut off. Ever. Every device always gets work. Trust is a factor in priority, never a gate.
• The orchestrator is never done — it hands out jobs forever and decides what has priority, from recency of a target's last result, the trust of who observed it, the provider-expected volatility (per-posting volatility aggregated over the provider), and so on.
• A low-trust result does not punish the device — it raises that target's priority, so the orchestrator is likely to send another actor to get the same result and corroborate. Trusted + fresh + stable targets sink in priority and are left alone; weak or stale ones rise and get re-checked.

The single stored value converges because trustworthy devices overwrite — not because anyone votes, and not because anyone is banned. The budget is device-work; the orchestrator spends it on the highest-priority targets.

Discovery, volatility & currency

• Discovery (find NEW postings) is its own leased target kind — crawl a listing/sitemap → child URLs. The orchestrator seeds + prioritizes the listing targets (from source config) and ingests the found URLs as new rows; the crawl is leased (the orchestrator never fetches — thin, datacenter-IP-blocked). It is not a volatility signal: every provider produces new postings.
• Volatility is per-posting — how a posting is expected to change over its lifetime (its closure/expiry is just another change, so it's covered). Aggregated over a provider's postings it becomes the provider's expected volatility, which is the usable input to the re-observation cadence: a provider whose postings change/expire fast gets its postings re-observed sooner.
• Currency, not deletion. Re-observation keeps the corpus current; a posting found gone is marked inactive — a normal observation result (one row, LWW) — and never deleted. We keep everything; we just know what's active.

Sizing — why 256 MB holds

• Tensor: KB–low-MB in RAM.
• Queue: a bounded DB query ("what's due"), not the corpus.
• Fold: moving average / median — arithmetic (can even run in the DB).
• Serving: one short call per device-online (tensor + lease + deposit URL together).
• Writes: device → Corbet S3 directly, falling back to B2 when the primary PUT fails; the orchestrator is not in that path. The object-store → CrateDB fold is the orchestrator's, but it is out of band (async/batched, on its own interval) — a device deposits and is done; it never waits on the DB.

So the control plane is "everything and nothing": pivotal leverage, almost no footprint. The only thing that could strain the box is lease-grant QPS at a huge fleet — fixed by handing out batches of targets, or fronting reads with D1. Not by a bigger box.

What this is NOT

• No observation history / per-URL vote tree. One row, the last one.
• No content gate, no per-write validation/enrichment, no fat ingest worker. Last writer wins; the only "check" is that the presigned deposit URL is valid, and it is object-scoped, so it authorizes exactly one deposit.
• No tensor in object storage/CDN. The orchestrator hands it out personally. The deposit store holds structured device deposits transiently; the independent raw store holds RS03-encrypted captures durably. Neither stores the tensor.
• No device is ever cut off / suspended / revoked. Trust only weights priority.
• The broker is not a control plane. Render hosts the frontend; the control plane (queue, leases, deposit minting, deposit store → CrateDB ingest) is the orchestrator.
• Axiom is not in the control loop. Telemetry is a sideways tap; the orchestrator never reads it back to schedule.

Where we are today (2026-06-09)

The thin orchestrator path is BUILT end to end (sporewright-server, alwaysdata) — phases 1-3 below are done and live:

• Its own priority queue from CrateDB (queue.rs, curl → :4200/_sql), ranked by sporewright::schedule::priority(staleness, trust, volatility), excluding in-flight leases; refilled into a private RabbitMQ work queue.
• Serves devices POST /lease → { lease, deposit_url, deposit_overflow_url, deposit_key, tensor } in one call, stamping leased_at.
• Mints object-scoped URLs itself (object_store.rs, pure-Rust SigV4): one S3-compatible store is the transient structured-deposit mailbox, and a second independently configured store is the durable RS03-encrypted raw archive. Corbet S3 is canonical for both object classes. B2 is a store-and-forward spool: structured deposits use the same logical key in either inbox; raw fallback lands under overflow/raw/, is streamed to raw/, read back and SHA-256 verified, and is deleted from B2 only after exact promotion. ingest.rs drains only deposits → CrateDB (the only corpus writer, async/batched) — one row per URL, LWW, no content gate. /observe + the DB lease-key write-through are removed (the presigned URL is the write credential).
• Axiom ops tap (axiom.rs): fire-and-forget lease/queue/ingest/planner telemetry to jobcache_telemetry; never read back for a control decision.

What's still the old shape (the remaining gap):

• The live device fleet still talks to the broker (api.jobcache.corbet.ch, Render) — there are no device clients for the new /lease+deposit protocol yet.
• The broker still does heavy per-write work and keeps a history tree (evidence + evidence_objects) on top of ad_observation_state.
• The orchestrator's tensor is still the M1 demo (hardcoded fleet) — handed out, but not yet fed by real device registrations. UPDATE 2026-07-02: the M2 routing design is LOCKED — see FLEET-ROUTING (archetype options, capability-only gates, trust as price+scrutiny under a freshness mediator, λ budget loop, occupancy guarantee); the M1 router is scheduled for deletion by that spec.

Migration (each phase keeps the corpus filling)

1. ✅ Orchestrator gets its own queue. CrateDB HTTP read path; local budget fold.
2. ✅ Orchestrator serves devices. One on-online {tensor, lease, deposit_url} call; leased_at stamping; RabbitMQ lease delivery.
3. ✅ Devices write by key → deposit by URL. The device deposits its one-row-LWW observation to Corbet S3 via the orchestrator-minted presigned PUT (object-scoped = the lease write key, since CrateDB has no per-lease RLS); the orchestrator ingests object storage → CrateDB out of band. B2 takes over when the Corbet PUT fails. /observe removed.
4. 🟡 Build the device clients + collapse storage. UPDATE 2026-07-02: the RUST device client is COMPLETE at HEAD (lease → scrape → RS03 archive → S3 deposit → POST /deposited notify; discover leases → POST /discovered with device-derived identity). The TS client covers fetch-page+deposit but not discovery. The storage collapse (drop the evidence history tree) is still open. (original scope below) Build the device clients + collapse storage. Write the TS + Rust device clients for the new protocol (lease → scrape → deposit). Drop the evidence history tree; keep one current row per URL carrying {value, raw-object locator, last_writer, content_hash} + the device-trust ledger. Migrate existing data.
5. 🟡 Cut over + demote the broker. UPDATE 2026-07-02: the interface strip is CODE-COMPLETE at HEAD — /api/devices/*, /api/tasks/enqueue, /api/planner/run, /api/coverage, device-control.ts, and planner/source-bindings.ts are REMOVED from jobcache-interface; Render is frontend-only (UI/API/MCP/sessions/drain) in code. Deploy (Render redeploy + fleet on the orchestrator transport) pending. (original scope below) Cut over + demote the broker. Move device traffic broker → orchestrator; strip device-control + planner out of jobcache-interface, leaving it the frontend (UI / API / MCP) only.
6. ⏳ Scale watch. If lease QPS strains the box, hand out batches or front reads with D1. Re-home the orchestrator only if genuinely forced — it shouldn't be.