Env Fetch Router — Cheapest-First Cascade
JobCache has to handle a mixed source landscape: friendly direct pages,
JS-rendered boards, and hostile detection-heavy boards. Paying browser-backed
rates for an HTML-only page would burn the unit economics. The router is a
device/runtime implementation of Env.fetch, not a central server scraper.
The order
- Direct fetch — cheap HTTP fetch from the executing Device or operator-owned harness. Works for simple sources.
- Firecrawl — managed scrape with JS rendering. Per-call cost. Works for JS-heavy boards.
- Browserbase — full stealth browser with proxy rotation. Highest cost. Reserved for hostile boards.
An Adapter run succeeds at the first tier whose output passes the per-adapter validity check (HTML size, expected selectors present, no anti-bot interstitial). Subsequent tiers are only attempted on failure.
The router source is jobcache/ingest/src/scrapers/router.ts; fetcher
implementations are raw.ts, firecrawl.ts, and browserbase.ts. Metrics in
scrapers/metrics.ts are emitted per tier so the cockpit can show how often
each tier was the one that produced the Observation.
Hostile-board behaviour
When a previously friendly board flips to JS-only or starts serving captchas, the router experiences a string of tier-1 failures and a sudden tier-3 win rate. Two consequences:
- The cockpit's source-matrix row turns yellow (latency went up; cost per ad went up) or red (Browserbase quota pressure).
- An operator can pause the source, accept the increased cost, or wait for the board to stabilize.
There is no automatic remediation — the router only chooses the next-cheapest path that works.
Why a cascade and not a per-source choice
A per-source mapping (linkedin → browserbase) breaks when LinkedIn relaxes or tightens its policy. A cascade is self-correcting: the cost per posting drifts with reality instead of with our last hardcoded decision. The cost panel and the source matrix together expose the drift to the operator.
See also
scrape-event-log.md— the append-only event table where each scraper attempt lands.jobcache/ingest/src/scrapers/— device/runtime fetcher sources backingEnv.fetch.wiki/content/studies/cockpit-vision/jobcache-cockpit.md— how the cockpit reads scraper-class usage.