Fleet routing — the tensor as navigator

Companion to CONTROL-PLANE (orchestrator internals: queue, lease, deposit, ingest) and SCRAPER-FLEET (host classes + rebuild requirements). This doc is the ROUTING design: how the sporewright tensor (see sporewright/docs/MODEL.md

• ROUTING-MODEL.md) governs which work goes to which kind of device, how trust is learned and spent, and how the system self-balances. The tensor is the navigator of this mess: every choice follows from a resolve over it, and the operator balances the system by writing facts and weights — never by adding routing code.

Implemented M6 evidence dynamics (2026-08-16)

The live path is corpus.operation.source.binding.task.environment. Every prefix adds a residual to the decision; a more specific address does not replace its parents. An executed outcome is written at the realized environment leaf, message-passes back up that exact path, and the retired task subtree is immediately compacted into its binding. The causal receipt preserves the original full address even after compaction. Long-lived state is therefore O(bindings × execution classes × observed dimensions), independent of ads, devices, or outcome-history length.

Two mechanisms prevent a once-correct routing belief becoming permanent:

1. Clock aging. JobCache applies precision(t + Δt) = precision(t) · 2^(−Δt / 48h) to every compacted evidence coordinate. The posterior mean stays fixed but its variance grows, so curiosity can revisit a class after networks, browsers, or portal policy may have changed. Recovery from CrateDB uses the same exponential weighting.
2. Observed drift. Each binding/class/dimension keeps two bounded online sufficient statistics: fast ← fast + 0.50(x − fast) and slow ← slow + 0.05(x − slow). After at least eight baseline observations, three consecutive same-direction gaps above 0.35 (binary outcomes) or 0.50 (log-scaled latency) confirm a local regime change. JobCache then discounts that exact coordinate's old precision by 0.20, applies the new outcome normally, and rebases the detector. Stable noise cannot repeatedly erase evidence, while a real reversal adapts much faster than the 48-hour clock alone.

Sporewright owns the generic address, message-passing, compaction, and exact discount operations. JobCache owns clocks, outcome normalization, thresholds, and the meaning of drift. GET /tensor exposes aggregate evidence_dynamics counters and confirmed drift traces, without reporter identities or an unbounded event replay.

Historical M5 decision and feedback path (2026-08-15)

The earlier M3 router was adaptive, but its learning path was not honestly layered Sporewright: JobCache folded device history and calculated per-(source, archetype) medians in ordinary Rust maps, then wrote the already aggregated answer into the tensor's source level. The device level was declared but unused. M4 moved the aggregation boundary into tensor operations; M5 replaces its temporary per-source tensors with one explicitly scoped tensor and lets every participant perform the final local realization.

The two directions now are:

decision: session policy/prior → source facts → job gates → archetype resolve
feedback: durable outcome → device fact → source median/sum → session median

This is “backpropagation” in the Sporewright sense, not neural-network gradient descent. An execution outcome is normalized into evidence; tensor reductions propagate that evidence toward broader contexts; the next forward resolve sees the changed landscape. A failure is therefore neither a participant error nor a permanent rule. It changes the probability/cost surface and the system can try something else.

The browser-visible observation surface is intentionally aggregate. GET /tensor and lease snapshots expose SESSION/SOURCE knowledge but filter raw DEVICE reporter cells and identities. Every participant folds that snapshot with a DEVICE scope containing its current circumstances. The scraping pill shows both the orchestrator decision and the participant's local realization. An executed failure follows device → source → session; a preflight protest stays at device and releases the optimistic lease without teaching the whole browser/desktop class. Playwright pins both loops.

One tensor, two independent questions

M5 cells use (layer, scope, option, dimension). A layer answers who owns and may revise this preference. A scope answers under which circumstances it applies. Known owner coordinates obey the session → source → job → device ordering; free context facets can specialize a preference without pretending to own it. DEVICE facts carry {source, device} scopes, SOURCE reductions select only descendants of {source}, and SESSION priors reduce the resulting source cells. Opposing-source tests and shared Rust/TypeScript golden vectors prevent cross-source pollution. No composite identities or temporary tensors remain.

The expected fleet shapes everything: a TON of browsers (every CareerVector client participates as a jobvector-device — one huge array collecting publicly available job information to help everyone get a job), some desktop-app users, some containers — mostly resource-constrained, a few big ones on owned hardware. The code must be sound and scalable BEFORE the fleet arrives; the fleet never precedes the code.

1. The tensor instance

Preference layers (coarse → fine): session ≺ source ≺ job ≺ device.

• session — the privilege-scoped operator session (one, today); holds the operator keys and global weights. No BYOK in this product.
• source — a job board / binding. Learned per-source facts live here (reduced up from device measurements): per-archetype reliability + latency, cors_open, render_required, antibot, expected volatility (change timescale T_s).
• job — one target (one ad URL, one crawl). Requirement gates land here.
• device — the finest tier. Holds per-device cells: capability attestations (corroboration reports) and raw measurements BEFORE the median reduce lifts them to source. Devices live on the context axis, never the option axis (ROUTING-MODEL §5): the population is unbounded, churny, and weak-dominated; which devices are connected is telemetry, not a routing input.

Options = archetypes — kinds of execution capability, self-declared at registration and corroborable:

2. The dimension vector (this product's fill)

Judgement dims (costs, lower = better): financial — the CLOUD cost a unit of work causes (client CPU is 0 by doctrine, §3); latency; reliability — the learned error-rate deficit per (source, archetype), median-reduced from device measurements.

Gate dims (+∞ values; capability is the ONLY wall): priv:js / priv:render, priv:cors — learned per source: a browser attempt on an unknown source is itself the probe; success writes cors_open=yes, failure no (the failure is a datapoint, not an error), priv:ip:residential, priv:antibot:<family>, priv:geo:<cc>.

Budget dims (λ shadow prices, weight = price): budget:b2-classB, budget:b2-classC (2,500/day caps), budget:cratedb-sql (semifree), budget:axiom-ingest (semifree). Per-source POLITENESS is deliberately not a λ pool — it is a rate, and rates ride the edge-local TokenBucket admission gate at lease-grant time (the §6 trap in ROUTING-MODEL: a slack daily quota never prices a rolling-window rate).

3. Cloud vs client — the economic doctrine

Device usage is free; cloud quotas are not. B2 is metered, the orchestrator and CrateDB are semifree. Therefore the algorithm conserves cloud and spends client compute freely:

• Occupancy guarantee: every device is always kept busy (some classes are ephemeral — an idle browser tab is capacity evaporating). When nothing pressing is feasible for a device, it gets FILLER: re-mapping work — re-observations of the least-recently-observed targets its archetype can fetch, and corroboration. Filler is infinite by construction (staleness regenerates) and is exactly what builds trust and keeps the corpus correct.
• Cloud-side costs per unit of device work (a deposit = B2 PUT + GET + DELETE + CrateDB COPY; a lease = CrateDB stamps) are the λ-priced resources; as a pool saturates, its price rises and the work mix self-shifts toward cheaper shapes (e.g. bigger lease batches, deferred low-value folds) without a rule being written.

4. Registration and live-scoping

A device self-announces on coming online (first /lease or POST /register): {device_id, session, archetype, capabilities[], budgets}. Consequences:

• Registration cells live in the orchestrator's RAM tensor at the device tier, live-scoped (MODEL.md §11): retained only while the device's lease/ heartbeat is fresh; pruned on expiry. One mechanism = GC + freshness + Sybil cost (a fake identity needs a live, renewed lease).
• Durable identity + trust persist in the CrateDB devices table (a returning device resumes its earned trust; the tensor is a lens, the DB is the clock).
• The refill loop only routes work to archetypes with live population — a 3am browser drought re-routes browser-feasible work to containers on the next refill tick, automatically.
• Registration is runtime and self-driven; HOW a host is deployed is a separate infrastructure concern (MODEL.md §10). Deployment ≠ registration.

5. Assignment — the lease decision

For device D (archetype A, trust τ) calling /lease:

1. Feasibility — resolve at the target's job cursor drops every archetype with a +∞ gate; a target is offered to D only if A survives. Capability is the only wall. Gates alone already reserve the scarce classes: an antibot, residential-gated portal is only ever feasible for desktop/ container-render, so the "keep the high-value devices for the few portals that need it" property is structural.

2. Value tiers — correctness outranks coverage. Correct information has the highest value — higher than new information. The map must be complete AND correct. The tier policy (all soft — no device is ever cut off):

   work kind	preferred τ	why
   first observation (never-observed target)	LOW τ first	there is nothing to corrupt — the alternative to a possibly-wrong datapoint is NO datapoint. Folding it auto-enqueues a verify for a high-τ device: low-τ breaks ground, high-τ certifies it. Net result: correct.
   verification (open verify-dispatch)	HIGH τ	one τ ≥ τ_hi verifier SETTLES a row (quorum 1 by policy); low-τ verifiers still count — their votes are never weighted (MODEL.md §11) — but the orchestrator demands a larger quorum k of them. Trust raises the quorum demanded, never the worth of a vote.
   re-observation of a settled row	soft-avoid LOW τ	the LWW hazard: we ALWAYS overwrite, so a lie here replaces correct data. Assignment-side preference only — never a content gate; if a low-τ write does land (sole populated archetype), the fold triggers a high-τ verify and the system heals.
   discovery (listing crawls)	mid/high τ	mints many targets; errors multiply.

   The inversion in row 1 is load-bearing: pointing low-trust devices at NEW ground and checking them with high-trust devices yields a correct DB; the reverse (low-trust re-observing good data) yields incorrect rows until somebody notices.

3. Occupancy — if nothing above is feasible for A, hand filler (§3). Politeness TokenBucket per source host is the admission gate on every grant.

6. Trust — a price and a scrutiny level, learned under a freshness mediator

The arithmetic lives in sporewright/trust (TS today; the Rust twin is #66 step 2 and part of this milestone): timeWeight(Δt) = 2^(−|Δt|/48h) and the symmetric pairwise update τ' = clamp01(τ ± base · τ_other · w(Δt)) with base = +0.02 on agreement, −0.04 on disagreement.

Freshness is a MEDIATOR, not a gate — it scales how much an agreement/disagreement between two observations of the same ad means:

One extension to the relocated math (the only new term): the disagreement branch is additionally discounted by the source's learned volatility — Δτ_disagree = −0.04 · τ_other · w(Δt) · (1 − vol_s) — so a known-dynamic source cannot burn the fleet's trust.

Deathloop dampers (the two failure modes named in review):

• Ever-changing pages: the churn window (nextChurn: 6 changes in 24h → 24h cooldown, already pure math in sporewright/trust) + the volatility discount + re-observation cadence following T_s — a hot page gets watched calmly, not fought over.
• Low-trust spirals: per-row open-verify budget (max k outstanding verifies), verify TTL, and the quorum policy above — verification always terminates.

Disagreement is a bug detector before it is a liar detector. Starting out, a "liar" is almost always an imperfect adapter. Disagreement events are clustered by (source, adapter_id, adapter_version) and emitted to Axiom as adapter_disagreement_cluster — a fleet-wide trust dip on one source flags the scraping code, not the devices.

7. The budget λ loop (wired, at last)

The primitives are built and golden-pinned (budget.rs/budget.ts); this milestone wires the product loop in the orchestrator: the slow tick aggregates each pool's measured usage (reduce_sum feeders: B2 op counts from lease-mint/ingest, CrateDB statement counts, Axiom emit counts), advances λ ← max(0, λ + α·(usage − cap)) single-writer, and publishes prices as the budget-dim weights — so resolve already carries λ·usage and load self-rebalances (Story 1 of ROUTING-MODEL, finally live). Politeness rates stay on the TokenBucket admission gate.

8. The handed decision — what /lease returns and why

/lease bundles {lease, deposit_url, routing_trace}. The trace contains the complete Sporewright receipt for the exact addressed decision. A participant folds that shared decision with its present circumstances. Fresh local negative reachability facts use an ageing curiosity window: they prevent immediate repetition, then reopen the source for trial. A local protest echoes the receipt, releases the lease with preflight=true, and is not projected upward; only an actual attempt becomes durable routing evidence. Successful deposits and executed failures retain the receipt so the selected class, decision revision, and contribution origins remain auditable after the tensor advances.

9. Scale (a ton of browsers is the design point)

• Per-archetype work queues (cv.work.<archetype>), purge+refilled each tick from the DB ranking — the queue mirrors the DB, never accretes.
• Browser-mass QPS: batch grants (N small targets per lease for browsers), heartbeat = lease TTL (no separate keepalive chatter), registrations are RAM-cheap cells + one devices upsert.
• If lease QPS ever strains the 256MB box: front the tensor-slice reads with D1/CDN (immutable-clock URLs) and grow batch sizes — never a bigger box (CONTROL-PLANE sizing doctrine).
• The tensor stays KB–MB: ~5 archetype options × O(10²–10³) sources, sparse; device-tier cells are live-scoped and pruned.

10. What this deletes

The M1 demo: router.rs's device-as-option axis, its hard priv:trust gate (both superseded — ROUTING-MODEL §5/§3C), the hardcoded 5-device fleet in main(), and the decorative lease:probe tensor field. The schedule module (staleness/trust-deficit priority, saturation→budget band) stays — it was always the doctrine-clean part.

11. Implementation order (each step GHA-gated; golden vectors pin cross-core)

1. sporewright core: Rust twin of trust (+ the volatility-discount extension in BOTH cores + golden vectors) — submodule commits + pin bump.
2. Registry: register/heartbeat-on-lease, live-scoping, devices-table trust persistence.
3. Archetype tensor: options/gates/learned-facts declared as TensorConfig; refill routes each due target to its best POPULATED feasible archetype (per-archetype queues).
4. Assignment policy in /lease: value tiers (§5), verify-dispatch enqueue on low-τ first-observation folds, occupancy filler, politeness buckets.
5. Trust learning at fold: pairwise events from prior-vs-incoming (device, content_hash, Δt), volatility estimate, churn cooldown, the adapter-bug Axiom cluster.
6. Budget λ tick + usage feeders + price publication.
7. Kill the M1 demo (§10) and hand the real tensor slice in /lease.