Observability — Push to Vendors, Pull /metrics for Cockpit
JobCache has two distinct observability paths. One pushes structured logs and traces to vendor dashboards (Sentry, Grafana Cloud) for incident forensics. The other exposes /metrics in Prometheus text format so the cockpit can read counts and rates without coupling to any vendor query language.
The push side lives in jobcache/shared/src/observability.ts, logger.ts, metrics.ts, and runtime-health.ts. Every service imports @cv/jobcache-shared and bootstraps the same observability before its HTTP routes go live.
What we push
- Sentry — uncaught exceptions, structured error events. Used for incident triage when something blows up in production. Sentry is a sink, not a source: the cockpit never reads from Sentry.
- Grafana Cloud (logs/metrics) — long-lived series for trend analysis, breaching alerts, and post-incident retros. Push via the OTel pipeline.
These are the dashboards an operator looks at during an incident, not the dashboards that surface the incident in the first place.
What we pull
Every service mounts a /metrics endpoint that returns Prometheus text format. The ops API and the status API both call this endpoint to build their snapshot panels:
- Postings counters (
jobcache_postings_inserted_total,jobcache_postings_duplicates_total) - Scrape counters per adapter and per source
- Runtime health (memory, latency p95)
parsePrometheusSums in @cv/jobcache-shared does the parsing. The cockpit reads from /metrics, not from CockroachDB, for these counts because Prometheus is much cheaper and avoids running aggregation queries against the hot store.
Why both
The push stack is for humans investigating after the fact. The pull stack is for the operator dashboard seeing the current state. They are different problems with different latency tolerances and different cost profiles. Trying to use one for both fails: Sentry is not cheap enough to be a real-time dashboard source, and Prometheus text is not searchable enough to be an incident archive.
See also
jobcache/shared/src/observability.ts— push-stack bootstrap.jobcache/shared/src/metrics.ts— Prometheus counters definition.jobcache/wiki/content/api/README.md— the/metricsendpoint as part of the public API.wiki/content/studies/cockpit-vision/jobcache-cockpit.md— how cockpit panels consume/metricsparsing.