Scrape Event Log
scrape_events is JobCache's append-only operational pulse. Device runtimes and
trusted operator-owned harnesses report one row per leased scrape attempt through
the control plane; the ops API reads from it to power the ingestion tape, the
source matrix rollup, and the recovery flow.
The table lives in CockroachDB. The schema is in jobcache/migrations/add-scrape-events.sql and the writer helpers are in jobcache/shared/src/scrape-events.ts.
What each row records
- Source identifier (
linkedin,jobs.ch, etc.) - Scraper class (
raw,firecrawl,browserbase) - Start and finish timestamps
- Outcome (
success,error,partial) - Counts: postings inserted, deduped, skipped, extraction failures
- Error category and HTTP status, when applicable
- Optional schema-drift flag
The row is the entire memory of the attempt. Nothing is mutated after insert; corrections happen as later rows that the cockpit overlays.
Three consumers
| Consumer | What it reads | Why |
|---|---|---|
Ingestion tape (/changes/stream) |
Tail of recent rows via SSE | Live "what just happened" surface |
Source matrix rollup (/source-matrix) |
Aggregates over the last hour and day per source | Green/yellow/red per-source state |
Recovery flow (/sources/:id/recover/rerun-last-failed) |
Most recent failed row for a source | Knows what to re-execute |
The status worker reads only sanitized aggregates from this table — it never sees individual scrape attempts.
Append-only on purpose
A row in scrape_events is a fact. Rewriting it would erase the operational history that the source matrix and the cost panel depend on. If a row is later understood to be misleading (e.g. a Browserbase quota hit that looked like a target-board failure), the correct response is another row — an operator note pinned to the source — not editing the original event.
This pairs with the operator-actions audit table (operator-actions.md) which is also append-only: events and recovery actions form a complete forensic trail for any incident.
Retention
No automatic purge today. CRDB rows are cheap and the cockpit's aggregate queries are bounded by time windows, so growth is acceptable. When volume becomes a concern, the retention job will roll up older days into a daily summary table and drop the per-attempt detail.
See also
jobcache/migrations/add-scrape-events.sql— schema.jobcache/shared/src/scrape-events.ts— writer + read helpers.operator-actions.md— the sibling append-only audit table for recovery actions.jobcache/ops/api/src/routers/changes.ts— SSE tail used by the ingestion tape.