Scraper Fleet

The collection side of JobCache. CONTROL-PLANE is the orchestrator internals, FLEET-ROUTING is the M2 tensor-routing design (locked 2026-07-02: archetype options, trust mechanics, budget λ, occupancy) (tensor + priority queue + lease + ingest); this doc is who scrapes, where the code runs, the contract between a scraper / the orchestrator / its S3-compatible deposit and encrypted-raw stores, and the requirements to rebuild it.

The scraping machinery is currently in ruins: the Render-hosted planner suspended on its free-tier monthly cap (2026-06-10), starving the whole fleet of leases; the orchestrator on alwaysdata is alive but burning B2 with a 30s LIST poll; the corpus is frozen at 7,498 ads. This spec is the target to build back to.

1. Organizing principle

Classes are defined by what CODE can run on the host — not IP, not trust. The dividing question is always: can this host run a whole (headless) browser, or only a lightweight fetch + parse? — plus, for hosts on a person's laptop, is it a good citizen (battery / connectivity / thermal)?

One scraping engine, shared across every class (same doctrine as CareerVector CLAUDE.md §23 "one engine, many hosts"). Adapters, method-chain, and method-learning are identical everywhere. Per-class differences are only (a) how it fetches (curl vs headless vs CORS-relayed) and (b) the citizen guardrails.

2. Host classes

NOT scrapers: phones (android/ios — never), personal dev machines, Render (frontend only; no collection, no orchestration). The android/ios/cellular/ battery/thermal fields in device-contract.ts are mis-attributed — the battery/connectivity/thermal fields belong to the desktop-app-on-a-laptop class; the phone platforms are vestigial (→ deprecate).

3. Capability axes (the real division)

1. Tier-1 vs Tier-2 — what code the host can run. Tier-1 = curl + schema.org JobPosting JSON-LD (~80% of CH boards; any host). Tier-2 = render/headless for JS-rendered/hostile boards; routes only to hosts that can run a whole browser.
2. CORS — the browser app can't fetch arbitrary cross-origin pages; capped to CORS-permitted targets or a relay. This is its defining limit.
3. Operational guardrails — the desktop-app-on-a-laptop reality: don't scrape on battery, survive intermittent links (work resumable/idempotent), thermal back-off. These are the power_source/battery_state/network_type/thermal_state fields, reassigned to this class.

4. Requirements

Testable. "Done" = acceptance met and observable in Axiom.

• REQ-1 — Pipeline works, now. A target leased from the orchestrator is scraped by a fleet container, deposited to B2, and folded into CrateDB as a corpus row, end-to-end. Accept: a fresh target goes lease → deposit → fold within one drain cycle; corpus row count rises; the path is visible in Axiom.
• REQ-2 — Orchestration: primary + PASSIVE failover. Orchestrator runs on alwaysdata as primary/default. A passive instance on local k3s (corbet-devops) promotes only when the primary is unreachable, and steps back when it returns. Accept: kill primary → k3s instance begins leasing within a bounded window; restore primary → k3s instance returns to passive; never two defaults; the k3s instance is never the default while the primary is healthy.
• REQ-3 — Ingest is NOT polling. Steady-state B2 Class C LIST ≈ 0. Ingest is driven by the orchestrator's own lease ledger (deterministic deposit keys it already minted) and/or a device "deposited" notify — never a bucket scan. Accept: empty inbox ⇒ zero LIST; daily Class C < 100 (cap is 2,500); no regression of the idempotent re-COPY safety.
• REQ-4 — Observability via Axiom. Every stage (lease, scrape, deposit, ingest-fold, error) emits to Axiom; an operator can answer "is scraping healthy?" — lease rate, deposit rate, fold rate, error rate, per-device activity, queue depth, B2 op counts. Accept: a live Axiom query shows these; data is confirmed flowing end-to-end.
• REQ-5 — Bootstrap on k3s. The starter fleet is containers on k3s (corbet-devops/Unraid), leasing from the alwaysdata orchestrator. Accept: ≥1 container device enrolled and actively leasing + depositing; corpus resumes growing.
• REQ-6 — Shared engine. One scraping engine across classes; container/edge/ desktop-app/browser shells reuse the same adapters/method-chain/learning. Accept: the container scraper imports the same engine the other classes will.
• REQ-7 — Citizen guardrails (design now, enforce per class). Desktop-app must not scrape on battery and must tolerate intermittent connectivity (resumable/idempotent deposits). Containers are exempt (always-on, AC). Accept: guardrails specified; enforced when the desktop-app class lands.
• REQ-8 — Frontend out of scope. The display frontend (Render / CF Workers) is explicitly NOT part of this work. The orchestration must not depend on it (see the planner extraction, §6).
• REQ-9 — No-cut-off + LWW preserved. No device is ever gated; trust only weights queue priority; one corpus row per URL, last-writer-wins. Failover/double-active must not violate this (double-leasing is at worst redundant scraping, which is safe).

5. Ingest contract — NO POLLING (REQ-3)

Current (wrong): ingest.rs drains every INGEST_SECS=30s via an unconditional ListObjectsV2 on deposits/ before checking emptiness (ingest.rs:103-108) ⇒ 2,880 Class C/day > the 2,500/day cap, forever, regardless of volume.

Why it's redundant: the orchestrator mints deterministic deposit keys (from ad_id) and stamps leased_at on every leased ad (ad_observation_state). The set of outstanding deposit keys = recently-leased, not-yet-folded ads — known for free from CrateDB. The LIST pays B2 to recompute what the orchestrator already owns (main.rs:241-247 deliberately chose "mark the lease, not store a key" — that choice is the bug).

Target (build both; ledger-sweep is the core, notify is the enhancement):

• (B) Ledger-driven drain (core, server-only, no device change): enumerate candidate deposit keys from the CrateDB lease ledger and attempt COPY on the deterministic keys; never LIST. A slow reconciliation pass (≤ daily) catches orphans.
• (A) Device-notify (enhancement, low latency): after a successful deposit PUT, the device pings the orchestrator (POST /deposited), which drains that one key immediately. Fits the model — the device already talks only to the orchestrator plus object-scoped presigned storage URLs.

Stopgap until landed: INGEST_SECS≥120 env on alwaysdata (read at startup, no rebuild) → 720 LIST/day, under cap. Buys time only.

Build state 2026-07-02: both (A) and (B) are CODE-COMPLETE at HEAD (see §9.1) — the deployed alwaysdata binary still runs the old LIST loop until the hub builds and ships the new closure, so the env stopgap remains the live mitigation.

6. ONE driver: the orchestrator (REQ-1, REQ-2, REQ-8)

There is no Render queue. There never will be again. (Enforced in code 2026-07-02: the interface no longer carries /api/devices/*, /api/tasks/enqueue, /api/planner/run, /api/coverage, device-control.ts, or the planner — pending Render redeploy. fact-tree.ts still mints verify rows into the orphaned tasks table on the drain path; harmless, tracked in §10.) Today the code carries TWO parallel work systems — (A) a Render relational tasks/leases queue filled by runPlanner (the only thing that creates new targets) and leased via /api/devices/lease, and (B) the orchestrator's RabbitMQ + ad_observation_state re-observation queue. (A) is abandoned/burned. runPlanner, planSourceBindingTasks, observeTaskFromDiscovery, device-control lease, GET /api/coverage, POST /api/planner/run, and the tasks/leases tables leave the data path entirely. The orchestrator becomes the sole driver of the whole loop.

The single orchestrator-driven loop (CrateDB + RabbitMQ + S3-compatible stores, zero Render):

                ┌──────────────────── orchestrator (alwaysdata) ────────────────────┐
  sources/      │  1. CREATE TARGETS: read sources/bindings → enqueue discover       │
  bindings ─────▶     targets; ingest crawled child URLs → new ad_observation_state  │
  (CrateDB)     │     rows (pending, never-observed)                                 │
                │  2. SELECT TARGETS: queue.rs ranks ad_observation_state by          │
                │     priority = staleness × (1+trust_deficit) × volatility           │
                │     (INCLUDING never-observed rows) → RabbitMQ                       │
   device ──lease──▶ 3. LEASE: POST /lease → {target, tensor, primary+overflow grants} │
   (k3s) ──deposit─▶ 4. device scrapes; PUTs to Corbet S3, B2 only on primary failure   │
                │  5. INGEST: ledger-driven drain → fold deposit store → CrateDB       │
                │  6. RE-SELECT: folded row gets fresh last_observed_at → sinks;       │
                │     stale/never-observed rise. Loop closes inside the orchestrator.  │
                └─────────────────────────────────────────────────────────────────────┘

What the orchestrator must ABSORB from the dead Render path (it already owns step 2, 5-partial, and the budget envelope derive_budget):

• Discovery seeding (planSourceBindingTasks → Rust): read sources/bindings, enqueue discover-urls targets under derive_budget's envelope.
• Discovery expansion (observeTaskFromDiscovery + loadKnownUrlHashes → Rust): a device leases a discover target, crawls the listing/sitemap, submits child URLs; the orchestrator dedups vs known and writes new ad_observation_state rows.
• Coverage read straight from CrateDB (tasks/ad_observation_state aggregates), dropping the curl /api/coverage hop.
• Device discovery on the orchestrator transport (currently discovery_not_wired, main.rs:171-176): wire /lease to hand out discover targets and add a POST /discovered endpoint.
• queue.rs must include never-observed rows (drop the last_observed_at IS NOT NULL filter) so freshly-discovered targets actually get leased.

Primary + passive failover: alwaysdata is primary/default. The same static-musl binary runs idle as a k3s pod on corbet-devops, promoting only when the primary's CrateDB leader-lease heartbeat goes stale, and stepping back on recovery. Active- active would be safe (LWW + no-cut-off + idempotent COPY) but wasteful — we run active-passive per REQ-2; the k3s instance is never the default while the primary is healthy.

7. Observability — Axiom (REQ-4)

Axiom is the ops tap (axiom.rs, fire-and-forget to a jobcache_* dataset; secrets/axiom.yml, org careervector-m3ma). The build VERIFIES it is wired and data flows, then fills gaps so an operator can see host health: per-stage events (lease / scrape / deposit / fold / error), counts/rates, per-device, queue depth, and store op counts (so a future LIST regression is visible). Never read back into the control loop (it's a tap, not a dependency).

8. Bootstrap + current state

• Bootstrap = containers on k3s (corbet-devops/Unraid), leasing from alwaysdata. Small share of eventual capacity, but they start the flywheel. First deploy: redeploy the jobcache-device pods pointed at the alwaysdata /lease (not the dead Render planner), on a rebuilt image (needs schema_org fix + handle_reparse; Rust builds go to the build-brain, never local; k3s rollout is the sanctioned path, not docker).
• Frozen since 2026-06-10: corpus 7,498 ads in CrateDB (269 MB / 3.3% of CRFREE); B2 jobcache-snapshots 61 MB / 0.6% of 10 GB; orchestrator alive but LIST-burning; pods on old image ec08c496; no device clients running.

9. Implementation plan (ordered; respects no-local-compute + no-docker)

1. ✅ Ingest no-LIST (REQ-3) — CODE-COMPLETE 2026-07-02, deploy pending: ingest.rs rewritten to the ledger-driven drain. /lease records the minted deposit key on ad_observation_state.deposit_key (add-orchestrator-discovery.sql; also boot-ensured by schema.rs); the sweep enumerates outstanding keys from the ledger (grace window + bounded per-key attempts + poison give-up), POST /deposited (device notify, key == proof-of-lease) folds one deposit immediately, and ONE reconcile LIST/day catches orphans. Telemetry reports attempted/folded/deleted/poisoned honestly. INGEST_SECS default 30→120.
2. ✅ Orchestrator absorbs discovery (REQ-1, kills the Render queue) — CODE-COMPLETE 2026-07-02, deploy pending: due-binding computation (freshness vs the new bindings.last_discovered_at/discover_leased_at ledger) + seed-url extraction ported into queue.rs build_discover_queue; the refill tick publishes discover targets at top priority under a backlog-derived budget (never-observed count replaces /api/coverage); /lease hands discover targets; POST /discovered bulk-inserts roles/ads/ ad_observation_state stubs with ON CONFLICT (ad_id) DO NOTHING as the entire dedup (the device derives normalized_url/ad_id/role_id via the frozen jobcache url_id math — identity stays in one crate, the orchestrator stays dep-lean); the last_observed_at IS NOT NULL filter is dropped (never-observed rows rank at maximal staleness). The JOBCACHE_API_URL//api/planner/run//api/coverage hops are removed (JOBCACHE_API_URL survives only as the optional frontend keep-warm). Render tasks/leases/runPlanner/device-control abandoned. NOTE: the TS orchestrator-device-client predates discover leases + /deposited (it idles on discover targets; its deposits fold via the sweep) — fine for the Rust-pod bootstrap, port when the TS class returns.
3. Resume the fleet (REQ-5): rebuild the Rust jobcache-device image at HEAD (gets the schema_org Failed-fix + handle_reparse + the discover/deposited transport; an orchestrator-only deploy no longer needs the broker env vars), deploy jobcache-device pods on k3s pointed at the alwaysdata /lease. Smoke: one container leases → deposits → folds → corpus +1, visible in Axiom. ORDERING: roll the device image BEFORE (or with) the new orchestrator if any pre-discover device build is still leasing — an old build pops a discover lease (top priority), can't parse it, drops it, and idles 60 s while the binding stays lease-blocked for 600 s: discovery gets eaten, not run. Today no device clients run at all (the ec08c496 pods predate the orchestrator transport entirely and never call /lease), so orchestrator-first is safe right now — but re-check before deploying.
4. Observability (REQ-4): add per-stage Axiom emits across the new loop + an ops query set; make fleet health legible/debuggable.
5. Passive failover (REQ-2): CrateDB leader-lease; deploy the passive orchestrator pod on corbet-devops.
6. Cleanup: POST /deposited notify (REQ-3 part A); deprecate android/ios from the contract; reconcile the RS02-read vs RS03-write reparse asymmetry; fix the failed_observation schema_id (jobcache.device-contract → cv.jobcache); fix RESOURCE-MAP/free-tier-infra DB drift. Any raw-retention deletion remains human-gated; the durable archive is reparse insurance, not ingest scratch space.

10. Cleanup / drift (tracked)

• device-contract.ts: deprecate mobile platforms; re-attribute battery/connectivity/ thermal to the desktop-app class.
• Ingest content gate is = 'success' only: a partial observation advances bookkeeping but never writes content/pointers. Defensible under whole-row LWW (a partial value would OVERWRITE a previously complete one), but it diverges from the broker-era fact-tree, which merged partial content field-by-field. Decide (<> 'failed' vs keep) before any adapter that emits partial matters.
• RESOURCE-MAP.md, free-tier-infra.md: Cockroach → CrateDB.
• Any doc calling Render a device/scraper or the orchestration host.
• Object stores: VersityGW at s3.corbet.ch is canonical for the transient deposit mailbox and durable RS03 raw archive. B2 is an independently scoped store-and-forward overflow: deposits fold from either inbox; raw overflow is copied to VersityGW, read back and SHA-256 verified before only the B2 spool copy is deleted. Canonical raw deletion remains outside this automated path and is human-gated.
• Missing or malformed transient deposits have a five-attempt budget persisted on the CrateDB lease ledger. Restarting the orchestrator cannot reset that budget. Retirement conditionally clears the stale lease and may delete only its exact deposits/ key; raw/ is never in the automated deletion path.