Architectures — Delivering Cockpit Data

The cockpit must compose three streams of evidence:

1. Snapshot — point-in-time aggregates (queue depth, counts, latency of last probe). Cheap to compute, expensive if computed on every page render.
2. Tail — events as they happen (process-request claimed, lease expired, provider 429). Continuous; either pushed or polled.
3. History — a rolling window of past events so the operator can answer "what happened in the last hour" without paging Axiom for every panel.

Each option below is a way to combine those three streams. None is free. The recommended option is labelled at the end as a recommendation, not a decision.

Constraints

Read every option against these constraints. Anything that breaks one of them is disqualified.

• Cloudflare free tier: 100k requests/day account-wide (CLAUDE.md §22). One ill-considered cron and we 1027 production.
• No periodic poller against deployed Workers for routine work (CLAUDE.md §22). Cockpit traffic is operator-only — it is small — but every "tick" multiplies by every viewer × every panel × every workspace.
• Browser is the preferred compute backend (CLAUDE.md §23). Cockpit reads should never push expensive aggregations into a server-side poll loop when the cockpit page itself can compute over a small delta.
• Existing primitives win. Reuse @cv/health, @cv/perspective-shared, the SSE fanout worker, Axiom EU-Frankfurt, and workspace_process_requests before introducing new infra.
• Bun + SvelteKit + Cloudflare Workers is the runtime. Architectures requiring node-only libraries, long-lived sockets in the page-render path, or non-Workers compute are extra work.

API layering — public/private as a process boundary

Locked. Cockpit data endpoints live on dedicated API workers, split by audience:

JobCache mirrors the same shape:

Why a process boundary, not a code-path boundary

The split is for DX and accidental-leak prevention. The public/private boundary becomes a deploy artifact, not a runtime branch:

• api.status worker has no admin-data bindings, no admin-route source code, no admin secrets. Leaking operator-only data through it is not a bug to write — it is a binding to add, which is impossible without touching wrangler.toml and reviewing intent.
• api.ops worker enforces admin auth at app-level Hono middleware. Every route is admin by default; non-admin routes are a deliberate exception.
• Type-system level: api.status routes return Health after projectPublic; api.ops routes return Health raw. Code-sharing happens through lib/perspective-shared/ and the new lib/cockpit-data/ (cross-workspace queries) — see "Shared code" below.

Compare to "one API worker with /ops/* admin-gated and /status/* public": that works, but a routing mistake or a forgotten middleware leaks. Process boundary makes the leak impossible by construction.

Why Hono

api.careervector is already Hono (post 204d6d86 rip /api/ prefix). Both new workers reuse the same toolchain, the same router patterns, the same testing approach. No new framework.

Shared code

Three shared libraries between api.status, api.ops, and api.careervector:

The shared code does not enforce the projection boundary by itself — the worker bundles enforce it. api.status's wrangler.toml simply does not bind to anything that returns operator-private data. Type-checking catches the "I imported a raw Health into a public route" mistake; binding absence catches the "I tried to query a binding I don't have" mistake.

Fall-through

api.ops reads product data sometimes — for instance, the "force stuck-request sweep" action calls the existing product endpoint at api.careervector. The mechanism is a Cloudflare service binding:

# api.ops/wrangler.toml
[[services]]
binding = "API"
service = "careervector-api"

Service bindings are worker-to-worker, do not count against external request quota, and avoid the round-trip through CF edge. api.status has no service binding to anywhere; it has nothing to fall through to.

Auth boundary

Admin login flow stays on ops.careervector.corbet.ch/login (current behavior); the resulting cookie is read by api.ops thanks to the shared parent domain. MCP-side admin needs a bearer token model — out of scope for the cockpit, but the slot exists.

Cost — no increase

Splitting hosts does not multiply requests. The cockpit's projected budget from the cost ceiling table below stays the same; requests just route to different workers. Service-binding fall-through is free.

A. Pure ISR polling (current direction, extended)

The simplest extension of what is shipped today.

• Status page: isr: { expiration: 60 }. Already deployed.
• Ops page: per-request render (no ISR — admin cookie). Each render re-runs readCareerVectorHealth() + new aggregate queries.
• Cockpit "live" feeling comes from the operator hitting refresh, plus a small setInterval in the browser hitting a JSON endpoint every 30s for delta panels.

Data flow:

[ops page render]
  → @cv/perspective-shared.readCareerVectorHealth() → D1 + probes
  → new readWorkspaceRollups() → D1 (workspaces, process_requests)
  → new readQuotaContribution() → AE GraphQL or D1 audit log
  All called via api.ops.careervector.corbet.ch
[browser → api.ops.cv.../tick every 30s] → small JSON delta → recolor rows

Costs

• Per ops viewer, per minute: 2 page loads × ~6 D1 queries = ~12 D1 calls/minute. Single operator → tiny.
• Per status viewer: cached at edge, ~0 origin requests after first per minute.

Pros

• Zero new infrastructure. Just SQL.
• Cache discipline preserved (ISR for status, no cache for ops).
• Easy to reason about: every query is a function of D1 state.

Cons

• "Realtime" is a polite fiction. The 30s delta tick is fine for most signals but bad for the event tape ("just now, lease expired on request X"). The event surface is the cockpit's centerpiece.
• Aggregate queries fan out across all workspaces on each tick. Scales linearly in workspace count; not catastrophic but unbounded.
• No history beyond what D1 currently retains. Axiom holds the deep history but is not queried.

When to pick: if the operator value of sub-minute event awareness is low and we want zero new moving parts. Probably the right starting point. Probably not the right end-state.

B. Reuse the SSE Fanout Worker, widened to cross-workspace

Already deployed for /api/workspaces/:id/changes/stream (per-workspace tail of workspace_sub_doc_ops, workspace_process_requests, workspace_agent_presence). Extend with a new endpoint:

GET https://api.ops.careervector.corbet.ch/changes/stream — admin-gated, tails the same three tables but unscoped to a single workspace. Cursor by ts_ms. 25s hold + reconnect, same as the per-workspace stream.

Data flow:

[ops cockpit page] → SSE connect to api.ops.cv/changes/stream
                    → Hono endpoint tails 3 tables by ts_ms
                    → emits typed events: sub-doc, process-request, presence
[panels] → reduce events into a live table
[snapshot panels] → still go through readCareerVectorHealth() called from api.ops

Costs

• Per ops viewer: 1 long-running SSE connection + 1 D1 polled tail every ~5s inside the SSE worker. CF free-tier sub-request limit applies; ~12 D1 reads/minute per viewer is OK.
• Status: unchanged. Status never subscribes to ops streams.

Pros

• Reuses an existing, tested pattern. The per-workspace stream code has tests and observed behavior.
• Sub-second event awareness for the event tape.
• Authorization is a one-line requireAdminForPage adaptation.
• No new storage, no new DOs.

Cons

• D1 polling at 5s on three tables works only because the tail is cursor-driven and bounded. Add a fourth table and the latency composes.
• "Live" cross-workspace tail by ts_ms can lose ordering near the cursor edge for high-write moments — three writes within the same millisecond from different Workers in different regions are not ordered. Acceptable for an operator event tape; not acceptable for causal reconstruction.

When to pick: when sub-minute event awareness has clear operator value AND we want to avoid introducing a DO. Recommended as the realtime layer. See "Recommendation" below.

C. New HealthAggregator Durable Object

A single DO (or one per region) that ui/mcp/realtime workers push events to. The ops cockpit holds a WebSocket to the DO; the DO fans out aggregated state.

Data flow:

[any worker handling a request] → ctx.waitUntil(env.HEALTH.fetch('/event', {body: …}))
[HealthAggregator DO] → in-memory rollup + periodic snapshot to D1
[ops cockpit] → WS to api.ops.cv/live
              → DO pushes snapshot + delta events

Costs

• Per emitting worker per request: +1 sub-request to the DO. At careervector-ui's current scale (~395 workspaces) this is manageable; at 100k workspaces it is meaningful.
• DO compute is metered. Idle the DO hibernates; the cockpit being open keeps it alive.
• New code: the DO, push helpers in workers, the WS handler.

Pros

• The DO is the natural place for cross-workspace state that does not belong in D1 (queue mix in the last 60s, provider 429 streaks, recovery event rate). Computing this from D1 every tick is wasted.
• WS push is genuinely sub-second.
• Hibernation matches the operator-only access pattern: cockpit closed → DO hibernates.

Cons

• New DO is the highest-friction option in this list.
• DO is a singleton (or per-region) — must reason about coordination if we deploy multi-region.
• DO outage breaks live cockpit; the snapshot fallback must remain.
• Couples every worker to the DO. Workers without service binding for HEALTH cannot push.

When to pick: if the cross-workspace aggregates we want (provider error rate, claim-class mix, recovery rate) cannot be cheaply computed from D1 on each panel render. Re-evaluate after measuring B.

D. Pull from Axiom (already wired)

The telemetry pipeline already exists for yellow/red tier workspaces. Ops cockpit becomes an Axiom dashboard client.

Data flow:

[browser, yellow/red tier] → Axiom EU-Frankfurt ingest
[ops cockpit panels] → POST to api.ops.cv/axiom/query
                       → Hono endpoint → Axiom APL query
                       → return rolled-up series

Costs

• Per panel per render: 1 Axiom query. Axiom free tier allows enough queries for an operator-only cockpit comfortably.
• Telemetry ingest is already paid for (build-flag gated today; in end-state per-workspace tier-gated).
• Query latency is real (Axiom queries are usually 200-800ms).

Pros

• Already paid for and already structured.
• Long-window history out of the box (event catalog already defines retention).
• Axiom APL is expressive enough for every query a cockpit needs.
• Status page can re-use the same queries for aggregate counts ("active workspaces last 24h" already wants this).

Cons

• Axiom queries cost time. Not suitable for the live event tape (sub-second), great for the rolling-window panels.
• Green-tier workspaces emit nothing to Axiom. The cockpit cannot use Axiom alone to surface "a green-tier workspace is now red operationally" — Axiom never heard from that workspace. D1 still has to power that path.
• Adds a vendor dependency to the operator path. Axiom outage means no cockpit. Acceptable if D1 fallback exists.

When to pick: for the rolling-window panels (last 1h / 24h provider rates, ingestion throughput) and historical drill-down. Recommended as the history layer. See "Recommendation" below.

E. Workers Analytics Engine

The Analytics Engine binding (ANALYTICS in CareerVectorEnv) is currently not_configured. Workers write events to AE; the cockpit queries via the GraphQL Analytics API.

Data flow:

[any worker handling a request] → env.ANALYTICS.writeDataPoint({blobs, doubles, indexes})
[ops cockpit panels] → GraphQL query to api.cloudflare.com
                       → aggregated counts/sums

Costs

• Writes are essentially free (built into Workers).
• Queries are GraphQL, count against Cloudflare analytics quota but not Workers request quota. Operator-only access pattern → tiny.

Pros

• Native to Cloudflare. Lowest friction to wire from a Worker.
• Designed for exactly this: short blobs, doubles, indexes, with high-cardinality breakdown.
• Free at our scale.
• Already on the roadmap (careervector-health.ts:148 flags it as not_configured, expected).

Cons

• GraphQL query latency 200-1000ms; similar to Axiom for slow panels.
• Schema is rigid (blobs/doubles/indexes structure). Less expressive than Axiom APL.
• Retention is 31 days on free tier. Adequate for cockpit, short for business analytics — Axiom keeps longer history.
• Adds a second analytics surface alongside Axiom. We end up with two: AE for system-internal Worker events, Axiom for browser-side yellow/red tier events.

When to pick: for system-internal Worker event counters that we do not want to put in the browser bundle path (e.g. "queued process_request inserted" emitted by the web worker on every job import). Recommended as the cheap counter layer for server-side-only events. See "Recommendation".

F. Hybrid

Mix the best parts of A, B, D, E. Probably the right end-state.

Snapshot panels      → A (D1 query through readCareerVectorHealth, extended)
Live event tape       → B (SSE Fanout endpoint widened to ops)
Rolling-window panels → D (Axiom queries) for browser-emitted events
                      → E (AE GraphQL) for server-emitted events
History drill-down    → D (Axiom, deeper window)

Each panel knows which source it queries; the cockpit composes them in the UI. The operator sees one cockpit; the cockpit reads from four sources.

Recommendation

Building stepwise:

1. Now. Ship cockpit on architecture A (extended ISR + small delta tick). Replace hardcoded infra rows in the current status page with snapshot.checks. Add queue-rollup and claim-mix panels that read straight from D1. Add workspace-rollups panel that computes per-workspace operational state from process_requests + updated_at. Adds zero new infrastructure.
2. Once that lands and we have a real operator feedback loop. Add architecture B (widen SSE Fanout to an admin-gated cross-workspace stream). Live event tape becomes sub-second.
3. In parallel with (2), wire Workers Analytics Engine. Provision ANALYTICS binding on ui/mcp/realtime; emit short counters for queued/claimed/succeeded/failed events. Cockpit gains a free rolling-window source for server-internal events (architecture E).
4. When yellow/red tier per-workspace consent lands (per telemetry-tiers.md), Axiom-backed history (architecture D) becomes useful for drill-down on a specific workspace within the operator's tier-allowed window.
5. Defer (C) — the new HealthAggregator DO. Re-evaluate only if measurements after (1)–(4) show the cross-workspace aggregates we want are too expensive to compute from D1 on each render. Likely not needed at the operator's traffic scale.

This recommendation prioritizes shipping operator value early on infrastructure we already trust, and growing into the richer sources only when they unlock something the cheaper options cannot.

The recommendation is reversible at every step. Architecture A alone covers a real operator's needs; everything after it is a nice-to-have that should be justified by operator pain.

Cost ceiling — rough envelope

The free-tier 100k requests/day cap is account-wide. The cockpit should add at most a few hundred requests per operator-hour, even during active use, to leave headroom for the actual product.

1350 / hour × peak 4 hours/day = ~5400 requests/day for one operator. ~5% of free tier. Acceptable.

Two operators online simultaneously: still ~10%. Three: ~15%. If we grow to a handful of operators we re-evaluate; we are not there.