Tier-2 Headless Ingestion (#68)

Decision (2026-06-07): JS-rendered + hostile boards are scraped by a self-hosted fleet of Playwright pods on k3s, called by a lean Rust device (thin client). No managed/external headless API — Browserbase is explicitly OUT (risk too high, 2026-06-07). The fleet is fully self-hosted. Builds + deploys ride the build-brain → boot.corbet.ch → k3s pipeline (see ~/.agent/topics/shared/fleet-build.md).

Capability ladder

A board flagged render goes to Tier-2. If a board defeats Tier-2's self-hosted stealth, it is flagged for manual review — we do NOT fall back to a paid external API. The lever is improving Tier-2 stealth, not per-page spend.

Why this shape

• Playwright, not raw Rust CDP. Hostile boards need best-in-class auto-waiting + the stealth-plugin ecosystem (Node/Playwright). chromiumoxide (Rust CDP) handles plain JS rendering but has a weak stealth story — wrong tool for the actual target.
• Browser as a service, device stays Rust. The device is static-musl Rust (broker lease-cycle, uniform fleet). Don't fold Node into it. The device sends a URL, gets rendered HTML/JSON-LD back — preserves the Tier-1 device shape and the build pattern. The browser scales independently.
• Fully self-hosted. Tier-2 carries all JS/hostile volume at zero per-page cost on the k3s Playwright fleet. No external headless API (Browserbase ruled out — risk too high). Boards that defeat self-hosted stealth are flagged for review; the lever is improving Tier-2 stealth, not paying per page.

Architecture

• Browser service (new k3s Deployment): a Playwright server exposing render(url, opts) → { html, jsonld, status } over the cluster network. Stealth config; concurrency-capped; bigger resources (Chromium ≈ 0.5–1.5 GB/instance). Image built via the build-brain — Nix dockerTools (nodejs + playwright-driver + chromium) preferred for fleet-consistency; pin upstream browserless/chromium as the fallback if Nix-packaging Playwright's browser paths proves too fiddly.
• Tier-2 device (new k3s Deployment): the same jobcache-device binary with a new render task class + JOBCACHE_DEVICE_TIER=headless, JOBCACHE_TASK_CLASSES=render. On a render lease it calls the browser service (cluster-internal URL), then runs the existing extraction + broker submit. No DB/credentials (same as Tier-1 — API-only).
• Capability routing (sporewright orchestrator): the device advertises DEVICE_TIER + TASK_CLASSES; the planner tags boards needing JS as render work and routes it only to tier-2 devices. Tier-3 is a per-board escalation flag when tier-2's success rate on that board drops.

Build / deploy (via build-brain — SAVED until keys / build-brain access)

1. Browser-service image → add a dockerTools.buildLayeredImage (nodejs+playwright+chromium) package to careervector/flake.nix; add a build_and_push … + ctr import line to infra/infra/scripts/fleet-build.sh.
2. Tier-2 device → reuses jobcache-device-image (the render task class lives in the same binary); a new k8s Deployment with the headless tier env + larger resources.
3. Deploy → k3s Deployments (browser service + tier-2 device) under jobcache/ingest/deploy/kubernetes/; capability routing shipped with the orchestrator (sporewright-server).
4. Both image builds and the k3s rollout are build-server work (fleet-build CronJob + ctr import + kubectl rollout) — deferred until keys/access per the standing "save build-server work" instruction.

Implementation order (code now; build/deploy when keys ready)

1. Device-side render task class (Rust, crates/jobcache-device) — thin client to the browser service. [code now]
2. Browser-service render endpoint + its flake.nix image derivation. [code now]
3. Orchestrator capability routing for render tasks (board JS-flagging + tier matching) in sporewright. [code now]
4. k3s manifests (browser-service Deployment + tier-2 device Deployment). [code now]
5. Build on the build-brain + deploy on k3s (Playwright-pod fleet + tier-2 device + the ≤512 MB lean device image for free micro-servers). [SAVED — keys/access]