Orchestrator design — working notes

Status: in-flight. This document captures what's been settled in design conversation between Julian and the agent. Sections marked OPEN are explicit gaps — do not invent content there.

Note (superseded mechanisms): the Borda seed-consensus and the matryoshka refinement-stack (§3a) described below as forward design are torched — the landed engine resolves one sparse sporewright tensor (resolveCascadeTensor), where seed consensus is a declared-presence quorum (count ≥ 2) and tier precedence is a weight over the full uncapped chain, not a layered refinement pass. Treat those specific mechanisms as historical.

Written verbosely on purpose so a fresh context can pick up.

1. Vocabulary

• Device. The unit. There is NO concept of "user" — there are only devices. A browser tab, a Tauri shell, a container, a CF Worker, a self-hosted Docker, a future agent runtime — all devices. Capability is declared per device. Work is executed by devices.

• Orchestrator. Single apex strategist. Flight-tower metaphor: makes hard decisions, sets standing orders, doesn't steer individual planes in real time. Not a queue manager. Not a per-job router. A POLICY ENGINE.

• Task primitive. A named kind of work, slow-changing. Examples: "greenhouse scrape," "LLM evaluation," "render PDF," "embed text." Each primitive has intrinsic source properties (does it need residential IPs, is it rate-limited, etc.) — these are properties of the source, not of the invocation. Primitive profile only changes when the source itself changes (LinkedIn tightens anti-bot, Browserbase publishes new pricing).

• Workspace. The scope for BYOK + policy variation. On CV side, workspaces vary — workspace A has Groq+Cerebras keys, workspace B has only OpenAI, workspace C has Browserbase too. Each gets its own policy per primitive. JobCache side has effectively a single global workspace (no per-tenant BYOK; server-provided keys; server-set ceilings).

• Policy. Cacheable on-device instructions, per (workspace × task primitive), pushed by orchestrator. Concretely: an ordered list of options the device can try.

• Chain. An ordered list of options. The current LLM cascade in lib/domain/src/ai.ts / chain.ts is the production prototype: a list of (provider, model, api key) slots tried in order with failover.

• Multi-dimensional chain object / gradient. The orchestrator's INTERNAL view of routing: a bundle of N chains, one per cost axis. Each chain is a partial derivative — sort the available paths along ONE axis with the others held fixed at the operating point. The bundle IS the gradient at that operating point. Lives in the orchestrator. Used to reason.

• Cost vector. Multi-axis annotation attached to each (device, method) pair. Axes include: financial, latency, reliability, quota burn, geo, more TBD. Tier (client / self-hosted / server) is a UX label for dashboards — NOT a routing primitive.

• Capability. What a device declares it can do. Includes runtime flavor (web tab vs Tauri vs container vs CF Worker), API keys held, hardware constraints (cpu/ram), network class (residential vs datacenter), reachable peers (P2P visibility).

• Protest. A device's "I can't fulfill this option" signal back to orchestrator. Triggers escalation.

2. The model in one paragraph

The orchestrator is a strategic policy engine. For every (workspace × task primitive), it computes a chain (an ordered list of options) and pushes it to devices in that workspace. Devices cache the chain. When work arrives, the device tries options in order — on failure, falls back to the next option in its chain. On exhaustion, it escalates to the orchestrator, which may extend the chain (e.g. reveal a precious commercial path) based on context — orchestrator has PRE-COMPUTED these branches like a chess engine, so the response is fast. The orchestrator learns over time from device feedback (latency, reliability, success rates) and adjusts chains accordingly. Devices also learn locally and may LOCALLY reorder their chain based on their own deltas from the global mean.

3. The three-level learning tree

Parallel to the jobs/ads/roles hierarchy CV uses elsewhere — lower levels can DIFFER from higher levels.

• Orchestrator (global) level.

  • Holds the global running means of MEASURED quantities (mean latency per service, mean reliability, etc.) — gentle exponential moving averages over time, not single-measurement flips.
  • Holds the global cost catalog (provider pricing, declared a priori — NOT learned because cost is known).
  • Generates per-workspace chains by combining global state + workspace overrides.

• Workspace level.

  • BYOK key configuration (which providers this workspace can reach).
  • Cost ceilings per upstream (Browserbase $30/mo, etc.).
  • Manual preference overrides if set by workspace owner.
  • These DO NOT change frequently — adding a new key or changing a ceiling is the typical workspace-level event.

• Device level.

  • Local deltas from global means: device E measures A's latency at 5ms vs global mean 2.5ms → records local Δ=+2.5 for A.
  • Device applies these deltas to RE-SCORE its workspace chain locally, and may locally REORDER it. The chain device E uses may differ from what other devices in the same workspace use.
  • Devices report measurements back periodically (batched) so the orchestrator can update global means.
  • Device-level reachability and capability: which P2P peers it sees, what hardware it has, what keys it holds.

The orchestrator pushes a workspace-level chain. The device applies its own local deltas to get its effective chain. The system as a whole self-corrects: global truth tracks aggregate reality; each device tracks its own deviation; both layers update gently to avoid thrashing.

3a. The refinement stack (matryoshka)

The learning tree above is the MEASUREMENT hierarchy. Orthogonal to it, chain RESOLUTION is a layered refinement stack: each layer takes the bundle from above, applies local refinement (narrowing / override / re-score), passes down. The number of layers is product-specific, NOT architecturally load-bearing. The pattern is invariant.

CV's current extension (6 layers, top → bottom):

Workspace + Primitive can FOLD into one layer for products without BYOK variation (e.g. JobCache). Other products could ADD layers (tenant within workspace, project within tenant, env within project) without changing the pattern. The math (Lagrangian over the gradient, EMA over measurements) operates over the full stack regardless of depth.

Matryoshka: each level peels back one dimension of context. Outer doll = most generic (global truth). Innermost = most specific (this device, this call, right now). The chain you ultimately execute is what's left after all dolls are unwrapped.

4. What's learned vs known

Cost is not a learning category. Reality of executed paths IS.

5. Chain failover semantics

• Device receives chain [A, B, C]. Tries A. Fails. Tries B. Succeeds. Reports success + measurements back to orchestrator.
• Device tries A, B, C. All fail. Escalates to orchestrator. Orchestrator consults its pre-computed extension: maybe D (a paid provider) was pre-decided as the next valid step → extends device's chain. Device tries D. Or orchestrator says "no extension, permanent failure."
• Options not in the device's chain DON'T EXIST to the device. Least privilege. Even if D exists globally, the device cannot independently decide to try it. The orchestrator must authorise it.
• Pre-computation: orchestrator already knows the full failover tree for the (workspace × primitive). Sometimes trivial (3-5 options). Sometimes bounded by computational complexity cutoff (when fleet × axes × options blow up).

6. The bijective channel

• Device → orchestrator. Reports:
  • Capabilities (runtime, hardware, keys, network class)
  • Reachability (which peers it sees, which servers it can reach)
  • Measurements (latency, success/fail observed)
  • Protests (cannot execute this chain step — reason)
• Orchestrator → device. Pushes:
  • The chain (ordered options to try for each task primitive)
  • Updates when chain changes (cache invalidation)
  • Extensions on escalation (an extra option appended on protest)
• The orchestrator NEVER probes the network directly. Its topology knowledge is built from device self-reports.

7. P2P transport is not authority

• Devices discover peers (bottom-up). The orchestrator learns topology from those reports.
• The orchestrator can include peer-routed options in a chain (e.g. "if you can reach peer Y, try sending the work to Y").
• But the authorisation to peer-route comes from the chain (which came from the orchestrator). Devices cannot peer-route around the policy.
• Mesh = transport mechanism, not a permission layer.

Locality flavours. "Submitter satisfies within own shell" is one of many locality concepts, not a single knob:

• Network locality (peer reachability, low-latency neighbours)
• Data locality (cached state already present here)
• Capability locality (this device holds the relevant BYOK key)
• Provider locality (recent warm path to a chosen SaaS — counterintuitive case: LLM "locality" is rarely about local compute, almost always about the call's warm path to a remote provider)
• Compute locality (CPU available here)

Devices discover most locality bottom-up and apply it as a local delta on top of the workspace chain. The orchestrator can ALSO encode locality-aware chain entries directly when it has signal (e.g. "if reachable peer Y, send to Y"). Both layers participate. Cheap on both sides; redundancy is fine.

Wheel with chords (topology). Julian's mental image: orchestrator at the hub of a wheel, devices at the rim, spokes = device↔orchestrator bijective channels, CHORDS = device-to-device direct connections (P2P via y-webrtc, LAN, mesh). Orchestrator sees only the wheel and learns about chords from device reports — its chord view is always stale. The full execution graph includes chords; the decision graph at the orchestrator is just the wheel. Chord-aware decisions are delegated to the rim, where reachability is real- time. Concretely: chains carry ABSTRACT peer-execute slots ("if a reachable peer matches capability X, route to peer"). Device fills the slot at call time based on actual mesh reachability. Reports back which peer was used so the orchestrator learns about peer effectiveness with a lag. Orchestrator owns POLICY (peer routing allowed for primitive X); device owns BINDING (which peer right now).

8. Multi-axis cost reasoning (the gradient)

• The full cost landscape: each (device, method) is a point in N-D cost space. N axes: financial, latency, reliability, quota burn, geo, etc.
• A chain is one ordering of paths along one axis (a partial derivative with all other axes held at their anchor values).
• The orchestrator's multi-dim chain object = bundle of N such chains = gradient at the operating point. Internal to the orchestrator.
• Solving the multi-objective problem is NP-hard in general. Standard technique: scalarisation (Lagrangian relaxation) — fold N axes into a single weighted scalar via workspace-set weights, optimise, find one Pareto-optimal point.
• The DEVICE gets the resulting ordered list. The orchestrator absorbs the complexity.

9. Existing prior art (the LLM cascade as proof-of-concept)

lib/domain/src/ai.ts, cascade.ts, chain.ts, providers-config.ts already implement this for ONE work kind (LLM compute), 1D (provider preference under implicit quality, anchored at cost=0).

• resolveChain(config, stageId, consumerId, capability) → ordered slot list
• callWithChain(chain, op, label) → failover loop with provider rotation
• L0 (stage defaults) / L1 (consumer overrides) / L2 (per-instance overrides) = the three-layer override hierarchy already implemented for static config
• Borda-count consensus for merging multiple consumer seed preferences
• Health probes per provider
• Audit log per call

Relationship to the new tree (resolved 2026-05-31): the L0/L1/L2 hierarchy is the existing WORKSPACE-LEVEL representation of one chain — a queue along one axis of the gradient. The new orchestrator/workspace/device tree subsumes it: orchestrator generates the workspace chain from global state + overrides; that chain still LOOKS like an L0/L1/L2 structure to downstream code. The current Borda+seed bootstrap is a "sorry excuse for an orchestrator" — manual approximation made before measurement learning existed. The device-delta layer is genuinely new. The orchestrator/global layer is genuinely new.

What's missing today and needs to be added in the generalised orchestrator:

• Multiple axes (not just provider preference under implicit quality)
• Learning (today's chains are static config, not measurement-adapted)
• More work kinds (not just LLM — also scrape, render, embed, …)
• Device-level deltas (today the chain is the same for every consumer)
• P2P transport awareness
• Pre-computed escalation paths
• Per-workspace policy generation from a global graph view

The LLM cascade is the PROOF that the chain primitive works. The new orchestrator extends the same primitive to all work kinds with measurement learning + multi-axis reasoning.

10. Adding new services = policy update, not code change

• Workspace owner adds Cerebras key → orchestrator regenerates that workspace's affected chains → broadcast → devices cache → use the new option where the cost math wins.
• CV ships a new adapter (new task primitive) → orchestrator extends its primitive catalog → workspaces inherit defaults → devices learn the new primitive next sync.
• New device tier comes online → device declares its capability → orchestrator considers it in chain generation for all primitives.

The architecture absorbs growth without restructuring.

11. CV vs JobCache mapping

Same model, different parameterisation.

Scope-ID generalization (decided 2026-05-31). The orchestrator operates on scope ID + primitive. The flavour of scope ID is product-specific: workspaceID in CareerVector, sessionID in JobCache, plus deviceID as a cross-cutting third member. Functions accept generic ID when they don't care which flavour; they accept workspaceID / sessionID / deviceID when they do. One library, one orchestrator instance, many ID flavours. The codebase refactor to ID-generic patterns is its own sweep, separable from the orchestrator build.

12. What this becomes (publishing scope)

If extracted as a library:

• Schemas + cached policy format → commodity, not the novel part
• The graph-routing engine (multi-axis chain derivation under workspace constraints + learning loop + device delta correction) → the genuinely novel IP
• Rust + TS via byte-equivalent schemas (same pattern as the adapter dock)
• npm + crates.io publishing targets
• Name: deferred — to be chosen together once design stabilises

13. Subsets that must remain functional during integration

The existing LLM cascade in lib/domain/src/ already serves production traffic. When integrating into the new orchestrator, the cascade must:

• Continue routing LLM calls without disruption
• Continue resolving BYOK keys for production workspaces
• Continue the L0/L1/L2 override semantics
• Continue Borda seed consensus
• Continue health probes and audit logging

The migration path is either:

• (a) Cascade rewired as the FIRST CONSUMER of the new orchestrator (clean separation; slower migration)
• (b) Cascade extended IN PLACE to become the orchestrator (faster but coupled) Decision OPEN.

Other production paths that must not break:

• workspace_process_requests table claim/lease mechanics (today's browser-claim + CF-Worker-fallback path) — the new model should initially shim on top, not replace
• Adapter dock contract (Rust trait + TS Zod mirror) — unchanged, but adapters might grow optional declarations (compatible_tiers, etc.)
• Existing cockpit data shapes (Zod schemas just landed) — orchestrator introspection adds new schemas, doesn't break existing ones

13a. Runtime placement and sizing (decided 2026-05-31)

Architecture split — BRAIN and NERVES.

The orchestrator container is the BRAIN: aggregator + decision engine. The CF nest (D1 + Workers + Realtime DOs) is the NERVES: chain serving + push fanout. Container does math; CF serves data. No double-duty.

• Container responsibilities: receive device measurements, update EMAs in RAM, write updated chains to D1, broadcast 1-byte invalidation signals via CF Realtime, fire-and-forget hourly EMA snapshots to Axiom.
• CF nest responsibilities: D1 holds workspace state + current chain state per (workspace, primitive); Workers serve chain reads from D1 (devices fetch on cache miss); Realtime DOs broadcast invalidations.
• Axiom responsibilities: cold tail — raw measurement events (sampled or aggregated), chain-decision audit log, trend dashboards. Container writes only, never reads back.

Sizing envelope (1M workspaces, ~5% active, ~50k devices online):

Placement: TBD by Julian. The shape targeted is a small always-on container (≤256MB RAM, ≤0.25 vCPU, ~100GB transfer/month). Back4App Containers is one candidate fit but the Back4App free slot is reserved for a personal VPN (the same shape happens to suit both). Single-instance SPoF in v1; warm standby when scale demands.

Tech: Rust.

• Static-musl binary in FROM scratch container (~5-10MB image)
• Cargo profile: lto=true, codegen-units=1, strip=true, opt-level='z'
• Preallocate hot state at startup (bounded MRU caches, bump arena for per-request scratch — no surprise allocations)
• Single tokio worker thread (or skip async; std::net + worker pool is plenty at 200 ops/sec)
• Restart recovery: rebuild hot state from D1 in ~30s; devices serve from local cache during the window

Wire frugality (decided 2026-05-31).

Target: stay well under 1GB egress per month to benefit mobile users (battery + data-plan) as much as it benefits the infra bill:

• Devices SAMPLE measurements (1-10% of normal traffic; anomalies and protests always sent in full)
• Binary encoding (bincode or postcard) for all on-wire payloads — 5-10× smaller than JSON
• HTTP/2 with gzip body compression on every connection
• Devices BATCH measurements (one send per ~5 min, multiple events per batch)
• Statistical sampling is fine because EMA convergence doesn't need every measurement; the orchestrator gets enough signal at 1-10% sampling

Realistic total: 250MB/month in + out combined under a 1GB cap (75% headroom). Outbound alone ~100MB/month after binary + gzip.

Caveats to verify on first deploy (whatever provider is chosen):

• Always-on (no spin-down on idle — orchestrator must hold hot state)
• Egress accounting (chain pushes to CF Realtime + Axiom snapshot sends)
• Persistent storage shape (or all-D1 write-back for EMA snapshots)
• Build pipeline + secrets injection conventions of the chosen provider

14. What's OPEN — not yet decided

• Tactical layer existence. Whether per-workspace queue/DO managers exist as a named architectural layer between orchestrator and devices.
• Pre-computation strategy. When does orchestrator pre-compute, cache, invalidate? Triggering events?
• Computational cutoff. When the state space gets too large, what is the fallback? Approximate / heuristic?
• EMA half-life. How "gentle" should the global mean updates be? Hours? Days? Per-axis?
• Device delta significance threshold. When does a device's local delta reorder its chain vs leave it alone (avoid thrashing)?
• Wire protocol for device ↔ orchestrator (push vs poll? WebSocket? Long-poll? Worker SSE?). Sub-question: when a device protests and orchestrator extends the chain mid-job, does the device sync-block on the extension RTT, or async-notify the user that another path is being tried? Interactive UX vs batch matters here. Constraint (Julian, 2026-05-31): OBSERVABILITY is the load-bearing driver — sysadmin should see all device↔orchestrator traffic in one ops dashboard. No double-channels. Working assumption (lean): orchestrator state as new op-catalog ops on per-workspace sub-docs, surfacing in existing op log / audit trail. Global state on a parallel channel still visible in ops dashboards.
• P2P transport substrate for cross-device handoff (y-webrtc? hyperswarm? both?).
• Security model (partial resolution 2026-05-31): Trust root = combination, but MOSTLY (a) trust-by-default with aggregation + outlier rejection doing the heavy lifting. Reasoning: when running on devices you don't control, exposure is unavoidable; lies drown in the mass at scale. (b) device attestation, (c) workspace-scoped vouching, and (d) reputation are nice-to-haves to layer on when cheap, but not foundational. Still OPEN: cold-start / low-N regimes where aggregation can't yet drown lies, rogue-submitter-forcing-routing prevention, and UCAN-style capability tokens for work-payload signing.
• Submitter vs executor (resolved 2026-05-31): Capability-bag model. Every device declares its capabilities; "executor" is just a label for any device with execute-* capabilities. Some declare submit only (today's MCP); some declare submit + execute-llm; some declare full sets. Roles emerge from capability sets, not baked into the model. Generalizes cleanly without cost.
• Policy schema. Concrete fields, format, versioning, invalidation.
• Reliability tracking mechanics. Where stored, how aggregated, how surfaced into chain updates.
• Bootstrap (resolved 2026-05-31): Device generates and persists a UUID locally — cross-tool (same UUID used by CV, JC, future tools — trackable everywhere in every sense of the word). On first contact with the orchestrator, device sends its UUID + capability declaration + workspace ID via the existing realtime channel. Where the UUID lives (browser localStorage / Tauri config / OS keyring / ~/.corbet/device-id) is a technicality; the principle is one stable UUID per physical device.
• Sub-orchestration / delegation. Whether orchestrator can grant bounded sub-orchestrator authority to a device.
• Cascade integration (resolved provisionally 2026-05-31): cascade is the workspace-level chain representation today; the new orchestrator inserts UPSTREAM of cascade chain population (orchestrator generates the chain that cascade then reads + executes against). Cascade's reading + failover logic stays intact. Exact migration cutover still open.
• Latency budget mechanism / per-invocation SLA. Same primitive invoked from MCP (batch) vs from a leader tab (human watching) — does the chain itself encode budget, or does the device pass a budget knob at call time? Closely related to L2 (per-instance override) survival.
• Capability vocabulary. Concrete tokens (fetch_plain, fetch_rendered, call_llm, etc.) sketched but not locked.
• Work-kind taxonomy. Concrete enum values (scrape, compute, render) sketched but not exhaustive.
• Library name. Deferred. Chosen together later.
• Publishing scope timing. Build-internal-first vs build-as-library from day 1.
• Latency outlier handling. One bad measurement shouldn't poison the EMA — bounded outlier rejection rules.
• Geographic routing dimension. How devices declare geo, how orchestrator uses it.
• Cost ceiling enforcement mechanism. Where ceilings live, how breach is detected, how breach affects chain generation.
• JobCache per-tenant fairness under shared keys. JC has tenant subdomains but BYOK=false. Single global policy, or per-tenant slices so one heavy tenant can't drain shared upstream quota?
• Claim_token + compute_lock integration (resolved 2026-05-31): Option B — orchestrator sits ABOVE these tactical primitives. Strategic vs tactical split: orchestrator answers "what to do where" (strategic chain generation); existing primitives answer "how to do it without waste" (claim_token = lease + heartbeat handoff for long-running stages on devices with unknown uptime; compute_lock = per-API dedup so we never ask the same thing twice). Both stay. Clean layering.
• worker_class lifecycle (resolved 2026-05-31): Stays as part of the tactical layer. The "tier as UX label" decision applies to orchestrator ROUTING (orchestrator doesn't key chain generation on tier); but the operational claim/lease layer still needs worker_class to know which kind of device can claim which kind of work. Coexists with capability declarations — worker_class is the coarse tier for claim semantics, capabilities are the fine-grained declarations for chain matching.
• L1/L2 override survival (resolved 2026-05-31): Extension, not collapse. L1 and L2 stay distinct. NEW config layers (Global, Workspace) wrap above; NEW runtime layer (Device) wraps below. Layer count is product-specific; pattern is invariant. See §3a (refinement stack).
• Borda + seed bootstrap (resolved 2026-05-31): Same algorithm, moves UP to the orchestrator level. Votes on the gradient (multi-axis slots) instead of 1D provider preference. Produces a multi-dim chain bundle.
• Exploration (resolved 2026-05-31): Every layer explores — orchestrator discovers TRENDS (macro patterns across the fleet), devices discover SPECIFICS (local-to-this-device phenomena), intermediate layers discover their own scale. Exploration runs in IDLE TIME — not just offline, but user-present-but-inattentive is the sweet spot (reading, designing — the device is available, no one is watching for results). Orchestrator owns the POLICY for when/where/how much. Resource-aware in BOTH directions: limits propagate UP (device's local quota state → orchestrator) and DOWN (orchestrator's view of shared resource state → device). Key ownership matters — Sarah's BYOK limit is her ceiling alone; a CV-owned shared key has a fleet-wide ceiling with different propagation. When a constrained resource is near its limit, conservation overrides exploration — don't experiment with what's nearly dry.
• Granularity vs caching tradeoff. Per-task chain (each invocation custom) is capturable but requires orchestrator escalation, defeating the cache that makes the system fast. The "mostly cached, escalate on protest" default needs to be named explicitly — and the escalation rate is a tunable, not a constant.
• Quota rationing inside a workspace. Workspace has 1000 calls/day on provider B. Distributed how across N concurrent primitives competing for it? FIFO, per-priority, per-primitive fair-share? Cost-ceiling enforcement in OPEN doesn't go this deep.

15. Reference points and prior art

• LLM cascade prior art: lib/domain/src/ai.ts, cascade.ts, chain.ts, providers-config.ts. Production code.
• CV CLAUDE.md §23 ("Compute backend — one engine, many hosts"). Existing 3-class worker model (browser / server / agent). This new model REPLACES that.
• CV CLAUDE.md §14 ("Singleflight Compute Locks"). Existing lease/claim pattern in workspace_process_requests. Substrate for the new tactics.
• jobcache/wiki/content/architecture/desktop-edge-runtime.md. Earlier proposal for Tauri-as-edge-device.
• jobcache/wiki/content/architecture/free-container-provider-inventory.md. Existing list of free container providers (Back4App, HF Spaces, etc.) — candidate devices for the new model.
• wiki/content/architecture/ADAPTER-DOCK.md. Adapter contract — task primitives plug in here.

16. Glossary of metaphors used in discussion

• Flight tower. Orchestrator metaphor. Sets standing orders, doesn't fly each plane.
• Chess engine. Orchestrator pre-computes failover trees N moves ahead — like a chess engine planning, with cutoff when complexity explodes.
• Shepherd. Orchestrator as caretaker — must be curious, learning, not stuck in failure modes.
• Differential derivation / gradient. Math view: chain = partial derivative along one axis; bundle = full gradient. Lagrangian relaxation for scalarisation.
• Cookbook. Not explicitly used, but the chain effectively functions as a recipe: "try this then that."