Ingest Operations

How postings get into the corpus, end to end, and how to operate the drain.

Pipeline at a glance

source-catalog.ts        Planner (hourly CI)        Broker (interface)        Device sinks         Corpus
(declarative sources) ─▶  POST /api/planner/run ─▶  /api/devices/* ─────────▶  lease + scrape ─▶   ads + ad_facts
 families/sources/         scans bindings,           validates lease,           submit                ad_observation_state
 adapters/bindings         enqueues discover +       dedups by url_hash,         observations          ad_field_state
                           observe tasks             writes the fact-tree        + discoveries

Five stages, one trust boundary: devices only ever talk to the broker API. They never receive DB, object-store, or provider credentials. The broker validates every submission (lease match, payload schema, url_hash dedup) and is the only writer of the fact-tree. A device is a scrape sink, nothing more.

The fact-tree, not a postings table

Live corpus data lives in ads (one row per posting URL) plus the fact tables ad_facts / ad_observation_state / ad_field_state. There is no postings table — anything reading one is wrong. The canonical count is:

curl -s "$JOBCACHE_API_URL/api/stats"      # → { total, bySource: [{source, n}, …] }

Discover → observe fan-out (the drain mechanism)

Discovery is stateless and uses dedup-as-cursor:

1. The planner enqueues a discover-urls task for a source's discover binding.
2. A device leases it, walks the source sitemap, and ships the whole URL set back (up to the binding's max_urls).
3. The broker hashes each URL, drops the ones it already knows, and enqueues the next bite of new scrape (observe) tasks.
4. Devices lease those, fetch one posting per active cycle, and submit an observation. The broker upserts it into the fact-tree → corpus grows.

Because dedup is the cursor, no device-side bookmark is needed: ship the full sitemap every run and the broker advances. This only works if max_urls ≥ the source's full posting count, or the device truncates before the broker can dedup.

Running a device sink

A sink needs exactly two env vars to reach the broker:

Politeness / pacing levers (all optional, defaults shown):

One worker fetches one posting per active cycle, so its per-board request rate is 1 / (activeMs + rand(jitterMs)) ≈ one fetch / 1.6 s, shared across all sources it has leased. The only way to speed the drain is to add more sink replicas — and that multiplies the request rate against the live boards. Keep it polite: the boards (e.g. jobup = TX Group) will rate-limit or block an aggressive crawler, and a block halts the drain. Three replicas (1.9 fetch/s aggregate) is the current, deliberately conservative, footprint.

Formats (all built; only k3s is live)

Tuning the drain ceiling — the four limiters

The corpus ceiling is gated by four independent caps. All four must be ≥ the target, or whichever is smallest truncates the drain. (This is the trap that once pinned the corpus at ~97: three caps were lifted and a fourth, hidden in the wire schema, still clipped the URL set.)

Freshness (how soon a binding is due to re-discover) is the DAY_MS-derived TTL in source-catalog.ts — currently 5 min, so the hourly planner always finds the discover binding due and enqueues the next bite.

After editing the catalog, materialize it into the DB:

bun run --cwd jobcache/interface catalog:sync     # idempotent; registers adapters into the `adapters` table
# then redeploy the interface so the compiled catalog ships

Planner

The hourly CircleCI jobcache_planner (.circleci/config.yml, on resource_class: corbet/unraid) hits POST /api/planner/run with the control token and only queues device work — it never scrapes centrally, so CI is decoupled from prod ingest and can go offline without stopping the corpus.

Manual tick:

curl -X POST -H "Authorization: Bearer $JOBCACHE_CONTROL_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"max_tasks":2000,"max_urls_per_binding":60000}' \
  "$JOBCACHE_API_URL/api/planner/run"

Diagnosing — the ops cockpit

https://ops.jobcache.corbet.ch reads the live control-plane tables (tasks / leases / devices / ad_observation_state / ads) — never the retired runs / scrape_events / Prometheus sinks.

Raw endpoints:

curl -s "$JOBCACHE_OPS_API/ingestion-health"   # queuedTasks, failedTasks, observationsLastHour, reconciled
curl -s "$JOBCACHE_API_URL/api/stats"          # corpus total + bySource

The reconciliation invariant cross-checks the cockpit's ads count against the interface /api/stats total (both off ads); it fires when they diverge beyond a small tolerance. That is the cheap guard that would have caught the original postings-vs-ads schema mismatch.

Healthy-drain checklist

• observationsLastHour > 0 and roughly equal to (replicas × ~2300/h).
• queuedTasks stays fed (the hourly planner re-discovers ~2000 new URLs/run until a source's full sitemap is exhausted, then that source plateaus).
• failedTasks small and mostly a known-paused source (e.g. jobs.ch is paused — robots disallows its detail paths).
• reconciled: true.
• Corpus total climbing toward the sum of the live boards' sitemaps (~45k with the current five sources; jobs-ch paused).

Scaling beyond the Swiss boards

The five hand-written TS adapters (jobup-ch + the Swiss boards) cap the corpus at tens of thousands. The path to millions is the 88 Rust adapter crates (crates/jobcache-adapter-*: Indeed, LinkedIn, Glassdoor, Adzuna, EURES + the major ATS families) — built but not yet wired into the live TS ingest. Wiring them (WASM host or the Rust device) is M4, and a deliberate decision, not an autonomous step.