CareerVector Cockpit — Concrete Views

This file describes the cockpit views for ops.careervector.corbet.ch (and the projection that lands on status.careervector.corbet.ch), assuming the recommended hybrid architecture from architectures.md and the tier model from telemetry-tiers.md.

It is a UI/UX brief, not a build plan. Views are designed for the operator's eye, not for users.

Layout — top level

┌──────────────────────────────────────────────────────────────────────┐
│  Careervector / Operations          [traffic light: SYSTEM]  [today]  │
├──────────────────────────────────────────────────────────────────────┤
│ Infra              │ Queue mix       │ Claim mix     │ Quota cost     │
│  D1     OK 18ms    │  q   ████░░ 47  │  brw  72%     │  ██████░░░ 51% │
│  R2     OK         │  cl  █░░░░░  8  │  srv  21%     │  account 24h   │
│  Realtime OK       │  run ██░░░░ 14  │  agt   7%     │  budget 100k   │
│  MCP    OK         │  ok  ████░░ 920 │               │                │
│  Typst  OK         │  err █░░░░░ 11  │               │                │
├────────────────────┴─────────────────┴───────────────┴────────────────┤
│  Event tape (live)                                          [pause]   │
│  10:42:13  ws/abc def..  process_request running → succeeded          │
│  10:42:09  ws/abc def..  lease expired, reclaimed by server           │
│  10:42:04  ws/xyz mno..  provider groq 429 → cerebras                 │
│  …                                                                    │
├──────────────────────────────────────────────────────────────────────┤
│  Workspaces                                       [filter: tier any]  │
│  ● ws/abc def..  yellow  q=3 r=1  last 2m  cv-profile…   provider 429 │
│  ● ws/xyz mno..  green   q=0 r=0  last 5h  evaluate…                  │
│  ● ws/qrs tuv..  green   q=0 r=0  last 1d                             │
│  …                                                                    │
└──────────────────────────────────────────────────────────────────────┘

Five regions: infra health, queue mix, claim mix, quota cost, event tape, workspace list. The event tape and the workspace list are the operator's working surfaces; the four header panels are the at-a-glance context.

Header panels

Infra check rollup

Source: readCareerVectorHealth() extended to do real probes for every binding it claims to check, not just presence.

Check Today End-state
D1 SELECT COUNT(*) round-trip Same. Latency exposed.
R2 Binding presence HEAD on a known sentinel key. Latency exposed.
realtime (rollup of all relays) Sum of per-relay probes Derived: degraded/down propagate up from the slowest/most-broken relay. Not probed directly.
relay-cf (CF DO) Service-binding fetch(/health) to the relay-cf worker Same. Latency exposed.
relay-deno (Deno Deploy) fetch(https://careervector-relay-deno.corbet.deno.net/health) from a CF Worker Same. Latency exposed. The probe uses the upstream *.corbet.deno.net hostname because CF Workers' fetch loops on in-zone CNAMEs; browsers still use the symmetric deno.relay.careervector.corbet.ch.
MCP Worker Not checked New: fetch(https://careervector-mcp.corbet.workers.dev/health). Latency exposed.
Typst service Not checked New: fetch($KOYEB_TYPST_URL/healthz). Latency exposed.
Analytics Engine Binding presence Same until provisioned; then 1 test write to a sentinel index.
JobCache API Already probed via readJobcacheHealth Cross-product check rendered in CV ops.

Each row shows: name, state badge (operational/degraded/down/ not_configured), latency in ms, last-checked timestamp on hover.

Status projection: the same five infra checks (no JobCache, because that has its own status). State badges only; no latencies. Hardcoded HTML rows in status/src/routes/+page.svelte are deleted — status renders directly from snapshot.checks.

Queue mix

Source: D1 aggregate over workspace_process_requests.

SELECT status, COUNT(*) AS n
FROM workspace_process_requests
WHERE updated_at > datetime('now', '-1 hour')
GROUP BY status

Five buckets: queued, claimed, running, succeeded, failed. Each row shows a sparkline bar (count) and the absolute count. Hover extends to "by stage" (radar/extract/enrich/evaluate/tailor).

Operator value: a sudden growth in queued with flat claimed means browsers stopped claiming. A growth in running with flat succeeded means a stage is stuck.

Status projection: none. Queue depth never leaks publicly.

Claim mix

Source: same table, breakdown by claim_class.

SELECT claim_class, COUNT(*) AS n
FROM workspace_process_requests
WHERE claimed_at > datetime('now', '-1 hour')
GROUP BY claim_class

Three classes: browser, server, agent. Shown as a stacked percentage. The CLAUDE.md §23 economic invariant ("browser preferred, server is fallback") translates to "this bar should be mostly browser during business hours."

Operator value: server share creeping up indicates browser starvation — users not having tabs open when their MCP-initiated work runs.

Status projection: none.

Quota cost

Source: ~/.agent/tools/cf-quota/check.mjs already exists for local use. Ops cockpit makes the same call server-side (admin API token, server-only, never bundled in browser):

node check.mjs --since=24h

Returns account-wide Worker request count for the last 24h. Cockpit renders as a budget bar: used / 100,000.

The single most important panel for keeping the system on free tier. Should turn yellow at 60%, red at 85%.

Status projection: none. Account-wide quota is operator-private.

Event tape — live

Source (recommended): SSE Fanout cross-workspace endpoint (architecture B in architectures.md).

Renders one line per event, newest at top, capped at ~200 lines. Event types and their renderings:

Event source Renders as
workspace_process_requests status transition ws/{slug} {kind} {old} → {new}
workspace_process_requests lease expiry ws/{slug} lease expired, reclaimed by {claim_class}
workspace_sub_doc_ops actor_class=system with provider key ws/{slug} provider {provider} {status}
workspace_agent_presence change `ws/{slug} agent {agent_id} {join
Infra state transition (probe goes from operational to down) infra/{check_id} {old} → {new}

For green-tier workspaces, {kind} is shown but the request body / result are not clickable. For yellow-tier, clicking opens a panel with stage attempt history and sanitized error category. For red-tier, the panel includes raw provider request/response.

The tape is the operator's "is anything happening" pulse and "did anything just break" alarm. Pausing is one click; pausing buffers events client-side so the operator can scroll back through the burst that triggered them to look.

Status projection: an incident-only tape — only entries where a check transitioned to degraded or down. Shows the last 24h of infra-state changes. No workspace ids ever.

Workspace list

The cockpit's centerpiece. One row per workspace, sortable by operational state and last activity.

Columns:

Column Source Notes
State Computed (see below) green / yellow / red traffic light
Slug workspaces.slug truncated Click → workspace detail
Tier settings.telemetryTier green/yellow/red badge, distinct from state
Queue Aggregate workspace_process_requests for this ws q=N r=M (queued and running)
Last activity workspaces.updated_at Relative time
Current issue Top-of-stack issue if state ≠ green Short string

Operational state rollup

The traffic light is derived from observed signals only:

State Trigger
red Any process_request failed in the last 5 min, OR a claimed request whose lease is more than 2× the heartbeat overdue, OR queued rows older than 10 min with no browser claim AND server fallback not triggered.
yellow Any process_request running for more than 60s, OR retried at least twice, OR claim_class flipping browser → server in the last 5 min.
green Otherwise.

Definitions are conservative on purpose. The operator should never see green when something is actually wrong; spurious yellows are acceptable.

Filters

  • Tier — show all / green / yellow / red tier workspaces. Useful for "show me everyone I can debug right now" (= tier ≥ yellow).
  • State — show only red / yellow workspaces. The default scan during incidents.
  • Last activity — last 1h / 1d / 1w / forever.

Workspace detail drill-down

Click a workspace row → side panel opens. Content varies by tier:

The workspace-focused Inspector uses GET /workspaces/:id/inspector as its API source and get_workspace_inspector as MCP parity. The dossier wraps the tier-gated workspace detail below with related Ops events, anomalies, journals, and operator audit rows. The live spec lives in ../../working-memory/ops-workspace-inspector-spec.md.

Always visible (green tier sufficient)

  • Slug, created_at, version, last activity.
  • Sub-doc clocks: settings, layout, order_state, cv-profile, cl-profile, jobs, notes. Helps detect "stuck-on-old-clock" pathologies.
  • Process-request queue: rows with status, kind, claim_class, attempts, timestamps. NOT result payloads, NOT error_message.
  • Operational state with the rule that triggered it (e.g. "red because request evaluate:xyz failed 3 times").
  • Quota cost contribution: estimated request count attributable to this workspace in the last 24h. Operator's bill, must be visible.
  • Process-request result.summary (typically: "found 3 matches", "extracted 5 fields").
  • Sanitized error category and stage where the error happened.
  • Provider chain that was used and which provider returned the final answer.
  • Sanitized recent activity timeline.
  • Live op tape filtered to this workspace.
  • Provider request/response excerpts.
  • Raw error messages and stack traces.
  • Cell origins drill-down with content snippets.

Recovery actions

These are operator-only POST endpoints in the ops worker, admin-gated. Each requires a click → confirm. None auto-run.

Action Effect When to use
Force-reclaim leases Clear claimed_by, reset status to queued for any rows with expired heartbeats. A claim worker crashed and the lease has not yet timed out by configured TTL.
Requeue failed request Insert a new process_request with the same kind and target, fresh request_id. A transient failure that the operator wants to retry without re-issuing from the user side.
Force stuck-request sweep api.ops calls the existing product endpoint POST workspaces/:id/process-requests/fallback on api.careervector via the API service binding, with bypassStaleAge:true. The cold-load primitive should have done this but didn't.
Toggle workspace state badge Manual operator override: pin to yellow ("watching") or red ("incident in progress"). Cleared on operator action or 4h auto-expire. Communicating intent across multiple operators in incident response.
(Future) Disable workspace Soft-flag that returns 503 from web API for this workspace until cleared. Containment for a workspace causing a quota spike or being actively abused.

Every recovery action writes a row to a new operator_actions audit log (operator id from session, action, target, timestamp, optional note). The audit log is admin-only and rendered as its own tab in the cockpit.

Audit log schema

Table: operator_actions (D1, append-only). See the migration at migrations/add-operator-actions.sql for the canonical SQL — column list, types, and indexes are intentionally not duplicated here so the migration file remains the single source of truth.

Canonical action enum. The cockpit emits exactly these action values (extend only by adding rows here and updating the writer in the api.ops worker — the column itself is unconstrained TEXT so old values keep reading clean as the set grows):

action target_type Meaning
force_reclaim_leases system Clear claimed_by and reset status='queued' on rows with expired heartbeats.
requeue_failed process_request Insert a new workspace_process_requests row with the same kind/target and a fresh request_id.
force_sweep workspace Call /api/workspaces/:id/process-requests/fallback with bypassStaleAge:true.
state_badge_override workspace Manual operator pin to yellow or red; clears on follow-up action or 4h auto-expire.
workspace_disable workspace Soft-flag that causes the web API to return 503 for the workspace until cleared (future).

Indexes cover the three common operator queries: recency (idx_operator_actions_ts), per-target history (idx_operator_actions_target), and per-action filter (idx_operator_actions_action).

Apply the migration (one-time, both local and remote D1):

npx wrangler d1 execute careervector --local  --file=migrations/add-operator-actions.sql
npx wrangler d1 execute careervector --remote --file=migrations/add-operator-actions.sql

Retention. Append-only by design. There is no UPDATE or DELETE path from the cockpit, and no automatic purge job. If log volume becomes a concern, add a separate retention job — out of scope for this table.

Cross-cuts (top-bar panels under "more")

A second tier of panels for less-frequently-needed views:

  • Provider chain health — per-provider 200/429/5xx counts over recent windows. Surfaces "Groq is 429-ing across all workspaces" immediately. Source: workspace_sub_doc_ops filtered to actor_class='system' with provider in actor_id, joined with Axiom for failure reasons in yellow/red workspaces.
  • Recovery activity — count of stuck-request sweeps triggered by cold-load in the last hour. Counts of requests recovered. Validates the architecture goal in topics/shared/careervector.md that request-scoped recovery is doing its job.
  • Compute lock contention — compute_lock rows by task type. Heat map of dedupe hits. Useful for noticing when the lock is preventing useful concurrent work or, conversely, when it is the bottleneck.
  • Audit log — operator_actions over time.

What status (public) gets

status.careervector.corbet.ch continues to be ISR-cached at 60s. The page becomes data-driven (no hardcoded infrastructure rows) and projects these:

Section Source Notes
Headline traffic light rollupHealthState(snapshot) Single state, big visual.
Headline metric workspaceCount and a refresh hint The current "N workspaces helping people find jobs" line keeps its intent; the number stays real.
Infrastructure checks snapshot.checks (public only) List rendered from data, not hardcoded HTML. Latencies hidden, only states shown.
Recent incidents Operator-authored incident log (new) Manually posted incidents and the auto-detected degraded/down transitions of the last 24h.
Last snapshot freshness Already shown; verify it is real The 60s ISR makes the displayed time bounded. Document the bound on the page.

What status drops (versus today's page):

  • The "Analytics — coming soon" section with — placeholders. Hide or remove. Do not ship placeholders (memory rule "no fake data" / "no silent defaults").
  • The hardcoded "Edge Workers Operational" row. If we cannot actually probe Edge Workers, the row does not exist.

What this leaves to follow-up work

  • MCP cockpit panels. The MCP perspective could have its own surfaces (most-used tools, recent agent sessions). Out of scope for first cut.
  • A11y pass. The cockpit is operator-only and the operator is always sighted today. When the team grows, this needs work.
  • Mobile. Cockpit is desktop-first by intent. Status is fine on mobile. Ops on mobile is not a near-term need.
  • Multi-region. If we deploy to a second CF region the cockpit must aggregate. Defer until we actually do that.
Source: wiki/content/studies/cockpit-vision/careervector-cockpit.md