Vision — The Cockpit
A cockpit is what an operator sits in front of when the system is alive. It shows what is happening now, what was happening a moment ago, where the attention is needed, and what levers exist. It is not a dashboard of yesterday's metrics. It is a windscreen, not a logbook.
CareerVector and JobCache each need a cockpit. They serve different audiences, so the views differ, but the underlying architecture is the same: every meaningful runtime event a Worker handles is observable, and every workspace has a single rolled-up traffic-light color the operator can scan in one glance.
Status is the cockpit's public face: enough signal to earn trust, never enough to leak per-workspace detail.
The two audiences
| Surface | Audience | Question | Privacy model |
|---|---|---|---|
status.careervector.corbet.ch |
Anyone (public) | "Is CareerVector working right now?" | Aggregate only. No per-workspace state ever surfaces. |
ops.careervector.corbet.ch |
Operator (today: Julian) | "What is the system doing? What needs me?" | Per-workspace state visible, but gated by each workspace's telemetry tier (see telemetry-tiers.md). |
status.jobcache.corbet.ch |
Anyone, including ingestion partners | "Is JobCache healthy? Is the corpus fresh?" | Aggregate corpus metrics. No source-specific detail unless the source is public-known. |
ops.jobcache.corbet.ch |
Operator + (future) business stakeholders | "Where is the corpus growing? Where is it stuck? Is unit economics OK?" | Source-level detail, cost, dedup ratios, scraper failures. |
The split is real, not aesthetic. Status and ops read from the same
underlying snapshot pipeline; they differ only in what the projection
exposes. projectPublic() in @cv/health already enforces this — drop
admin-only checks, drop details blobs, drop non-public metrics. The
vision extends the same discipline to live event streams: the same
event source, two projections.
What the CV ops cockpit must answer
These are the questions an operator should be able to answer in under
five seconds of looking. Each one drives concrete cockpit views in
careervector-cockpit.md.
- Is any workspace red right now? Why? The workspace list, sorted by operational color, with the dominant incident on the row.
- Is the request pipeline flowing? Counts by status (queued/claimed/running/succeeded/failed) cross-workspace. A queued-not-claiming spike points at browser starvation or fallback stuck. A running-not-completing spike points at provider exhaustion.
- Who is doing the work? Browser, server, or agent? Claim-class mix over the last 5/60/1440 minutes. CLAUDE.md §23 makes "browser by default, server as fallback" a load-bearing economic choice; the cockpit must show it holding.
- Are provider chains healthy? Per-provider 200/429/5xx counts over recent windows; first provider exhausted in a chain; chains where L0 fallback was triggered.
- Is the stuck-request recovery primitive working? Workspace cold loads that triggered a sweep, requests recovered, leases reclaimed.
- Is anything burning Cloudflare quota? Account-wide request count (today + rolling 24h) with a budget bar relative to the 100k/day free-tier cap (CLAUDE.md §22). One bad cron and we 1027 production.
- Did anything just break? A live event tape of degraded/down transitions, error spikes, lease expirations, and provider 429s with workspace and stage attribution.
The operator should be able to drill from any of these into a single workspace's process-request queue, then into a single request's history (claims, attempts, providers tried, outcomes). The cockpit is recursive.
What the JobCache ops cockpit must answer
JobCache is business-facing today by design — it owns the shared role/ad fact tree and pays the ingestion cost. The cockpit reads correspond to corpus economics, not user friction.
- Is the corpus growing? At what rate per source? Postings inserted (deduped) vs. duplicates skipped, by source, over recent windows. A source dropping to zero is silent failure today.
- Is extraction succeeding? Per-source ad → role extraction success rate, per provider used. Confidence distribution.
- What does each ad cost? LLM tokens spent per insertion, scrape request count per insertion, cache hit ratio.
- Are scrapers and sources healthy? Worker health, source-specific error categories (schema change, rate limit, cookie expiry).
- What is the corpus consumer pattern? Which CareerVector workspaces (or which workspace cohorts, anonymized) pulled which ads. Pull-through rate: ads that became workspace jobs.
- Inventory health. Total ads, fresh in 24h, expired, dedupe ratio, median freshness of currently-visible ads.
See jobcache-cockpit.md for the concrete views.
What status must answer (public)
Status earns trust if it answers these in five seconds for an anonymous visitor, and answers them truthfully:
- Is the system up? A single traffic light, derived from
rollupHealthState()over real probes. - How fresh is this answer? The "last snapshot" line on the current status pages is correct in shape; freshness should be a bound, not a hope.
- Are there any incidents? Recent incidents posted by the operator, the last 24h of automatic degraded/down events, mean-time-to-recovery.
- Has the system done work recently? Aggregate workspace activity in the last hour. Just enough to signal "real users are using this."
- For JobCache specifically: "How many companies are hiring through us right now?" The corpus count and freshness are the headline for an ingestion partner deciding whether we are credible.
Status must never leak: per-workspace state, per-source ingestion volume (JobCache contractual concern), provider keys, or queue depth (operators manage queues; users do not need to see them).
Why the operator needs realtime
A snapshot every 60 seconds is a logbook. The operator's job is to notice and intervene before users do. Three classes of incident require sub-minute detection:
- Provider chain collapse: when the first provider in a chain returns 429s and the cascade does not fall through (chain misconfiguration, or all keys exhausted). Users see slow or stuck tailoring; the operator should see the chain state changing live.
- Stuck-request recovery starvation: cold-load sweep should pick up expired leases; if it doesn't, work queues silently. The cockpit must show the recovery event count rising or flat-lining.
- Workspace going red: a workspace that just transitioned to a red operational state (stuck process_request, hard failure, provider exhaustion) is a candidate for proactive support. By the time the user complains on a chat channel, the operator should already know.
Status does not need realtime in the same way — public visitors do not intervene. A 60-second freshness floor is correct there.
Vocabulary discipline
The same color words mean two things in this product. Keep them distinct and the cockpit reads correctly:
| Term | Meaning |
|---|---|
| Telemetry tier (green/yellow/red) | Consent gate. What the workspace owner has opted in to share. See telemetry-tiers.md. |
| Operational state (green/yellow/red) | Observed system health for that workspace. Derived from process-request queue, recent errors, recovery activity. |
HealthState (operational/degraded/down/not_configured) |
The @cv/health schema's check state. The product-level rollup uses these four values. |
The operator reads "this workspace is red operationally, but only yellow tier — so I see the queue but not the LLM payloads." A workspace in red tier is unusually richly observable for as long as the time-limited consent stands.
What the cockpit is not
- Not Workspace Analytics. Workspace Analytics shows the workspace owner their own search patterns. The cockpit shows the operator the system's behavior across workspaces. They share no UI.
- Not Platform Analytics for commerce. Platform Analytics is the long-running aggregation used to infer market structure. The cockpit is real-time and operator-scoped.
- Not a logging UI. Axiom (already wired in
ANALYTICS-EVENT-CATALOG.md) holds the deep log archive. The cockpit can link to Axiom queries for drill-down beyond the live window; it does not replace Axiom. - Not a runbook. Recovery actions in the cockpit are bounded
primitives (force-reclaim a lease, requeue one failed request).
Complex investigations stay in runbooks under
wiki/content/runbooks/.