LLM Cascade — one resolver for every model and service call

The cascade is the single way every model or service call in CareerVector picks a provider, a model, and an API key. Importing a provider SDK directly, reading an env var inline, or building a stage-specific fallback list is forbidden because each of those paths fragments user-visible configuration and bypasses key rotation.

Storage keys and preference owners

The existing keys materialize three of the five preference owners: workspace → stage → consumer → instance → device. L0 writes STAGE, L1 writes CONSUMER, and L2 plus a call-local override write INSTANCE. WORKSPACE and DEVICE are first-class layers even while current storage does not populate them.

The tiers are preference owners (coarse → fine), not slot caps. There are no per-tier slot limits: each present (provider, model) option carries a route preference value (lower = better), and a per-tier precedence weight makes a finer override always sort ahead of a coarser one. The fallback chain is not a hand-tuned ranked list — it emerges from resolveCascadeTensor (lib/domain/src/cascade-tensor.ts) over one sparse sporewright tensor.

Each chain value is keyed by capability — not just "chat." Capability is a free scope facet, independent of ownership: a STAGE preference can differ for chat and embedding without inventing another layer. A stage with N capabilities has N independent option groups at L0. Resolution reads only the requested capability and returns the full objective-ordered set: every L2 option, then every L1, then every L0, by declared preference within a tier. A duplicate option that appears across tiers folds to its strongest (finest-tier) cell and appears once at its best position. Higher levels deviate; they do not replace the entire chain.

Consumers and seeds

Every consumer of LLM work declares itself in PIPELINE_STAGES with:

{ id, label, capability, seed: ChainSlot[] }

The seed is an ordered, ranked preference. Pick free-tier models that work for the task; the declared order of each consumer's slots becomes that option's route preference value. On first workspace access, buildInitialChains() groups consumers by capability, computes a declared-presence quorum consensus per capability (a (provider, model) option enters L0 when at least two independent consumers name it — a count, not a positional Borda score), and materialises L0 + L1 deviations into cloud_keys.chains. After that, chains live in storage and are edited by the user in the Models tab.

Consumers are the only place a stage author expresses "what I'd prefer." After seeding, all preferences live in workspace state, alongside the user's BYOK keys.

Resolution

resolveChain(config, stageId, consumerId, capability) → ProviderSlot[]

delegates to resolveCascadeTensor, reads only the requested capability, and orders the tensor's cells by tier precedence (L2 ≺ L1 ≺ L0) then declared route preference within a tier. A (provider, model) option that appears at more than one tier folds to its finest cell and surfaces once at its best position. It attaches keys via resolveKeyForCapability and returns the full objective-ordered slot list ready for callWithChain. callWithChain executes the caller's function against the first slot, falls through on errors, 429s, or critic rejection, and reports the executed provider for cell origin tracking.

Key sources

GET /api/chain returns the resolved chain for a workspace. PUT /api/workspaces masks keys (gsk_****cdef) on the way out and detects masked values on the way in so existing DB keys are preserved.

Recipes (canonical)

The four recipes for adding new AI functionality are documented in CLAUDE.md §8. In short:

1. New consumer in an existing stage: add a { id, capability, seed } entry to PIPELINE_STAGES, write the function in lib/domain/src/llm.ts, call it via resolveChain + callWithChain.
2. New stage: append to PIPELINE_STAGES (with ledKey, capability, consumers), add the xstate state, and provide an actor. The editor and labels pick up the stage automatically.
3. Non-pipeline call mapped to a capability: use resolveKeyForCapability(cloudKeys, capability, env, appMode).
4. New service provider: register in PROVIDERS with envVar, signupUrl, and capability. Adapters live in lib/domain/src/providers/.

Anti-patterns

BYOK and origin

The cascade is the only place where a workspace's BYOK key is read. The executed provider returned by callWithChain flows downstream into the op's actor_id (<provider>:<model> for system-class writes), which the dashboard projects into cell tints (see cell origins)). This is the chain that ties "which key paid for this answer" to "which cell looks like that on screen."

Why one cascade

CareerVector has a long tail of LLM consumers — extract, enrich, evaluate, tailor (multiple sub-consumers), translate, salary estimate, RADAR sweep, sentiment. Building per-stage retry logic guarantees the system will drift between them. The cascade collapses all of that into one resolver and one executor, with one user-visible config in the Models tab.