Work Queue + Singleflight — three concurrency layers, one stack

CareerVector coordinates concurrent work at three different granularities. They are layers of the same stack, not alternatives. Confusing them is the fast path to either burning the LLM budget on duplicate calls or losing a write to a tab race.

Layer 1 — workspace_process_requests

workspace_process_requests is the durable queue of stage-level work. A row encodes "for workspace W, run stage S on job J with input I, claimed by worker_class WC." Browser tabs, CF Worker fallback, and (in time) agent backends all claim rows from this queue.

claim_token plus a heartbeat lease is what makes one worker run a stage at a time. While the lease is live, no other backend can claim the row. The heartbeat lets long-running stages (multi-LLM-call tailor passes) keep their claim through unstable network conditions without forking the work.

request_id is deterministic for the stages where re-enqueue must not fork: hash(workspaceId, jobId, branch_path, stage). Two clients enqueuing the same tailor request collapse at the table's primary-key constraint, so the queue cannot double-up.

This is the canonical reference for "the MCP can submit work without a browser present" — that promise is implemented entirely by this queue.

Layer 2 — compute_lock

compute_lock lives a level deeper. Inside one stage execution, the worker may make several expensive third-party calls (one LLM call per custom column in evaluate, one Maps call per commute calculation in enrich). N tabs that all claim adjacent stage rows can still race the same per-cell LLM call. compute_lock dedupes that.

Protocol per expensive call:

1. Pre-flight check. If the column is already filled or the commute is already set, return without touching the lock.
2. Acquire lock. INSERT OR IGNORE on (job_id, task). Atomic.
3. Double-check inside the lock. Catch the race between pre-flight read and lock insert.
4. Compute. Make the third-party call.
5. Release lock. DELETE the row.

Locks have a 60-second TTL; expired ones are cleaned up on the next acquisition attempt, so a crashed worker does not block forever. The client treats { locked: true } as "someone else is on it" and waits for the realtime broadcast.

compute_lock is needed only when the work granularity is finer than the process_request row. When they align (one scrape per RADAR request), the lock is redundant — claim_token already serializes the scrape.

CVL's current tailor path has no compute_lock because its request_id is deterministic and the xstate tailor stage runs as one shot per target; the upper layer already serializes the call. If a future gesture regenerates a single CL section without going through full tailor, that call will need its own compute_lock keyed on (jobId, branch_path, section_id, 'tailor').

Layer 3 — opId envelope

opId is for the write side of the SDK, not for the work queue. Every op in @cv/mutations carries an optional opId (mixin OpIdEnvelope). The retry-safe SDK in @cv/workspace-client mints a nanoid for an op before its first send and preserves it across retries, tab reloads, and reconnect replays.

The server uses (workspace_id, sub_doc, op_id) as a dedup key in workspace_sub_doc_ops. A retried op returns the original commit clock without applying the op a second time. Without opId, a flaky network during a node.move would corrupt order state every time the user retried.

opId is mandatory for SDK paths that may retry; it is optional for server-internal callers, tests, and one-shot imports where the call site guarantees one attempt.

When each layer applies

MCP enqueues evaluate(job J)                        ← layer 1
   ↓
Browser leader claims process_request row           ← layer 1
   ↓
For each custom column:
   acquire compute_lock(job J, 'eval:<col>')        ← layer 2
   call LLM via cascade
   release compute_lock                             ← layer 2
   ↓
Stage writes patch via client.applyOp(...)          ← layer 3
   Server dedups by opId, commits, broadcasts

If you remove any one of these layers, an observable category of bug returns:

Anti-patterns

• "Just" using claim_token finer than a stage. The claim_token has a long lease; making it per-call collapses the lease design.
• Using compute_lock to coordinate work across workspaces. The lock is workspace-internal; the queue is the cross-workspace coordinator.
• Trying to dedupe ops with a request hash on the client. The server has the authoritative op-log; let it be the dedupe surface.

This three-layer split is the reason CareerVector can run the same workload from a browser tab, an MCP agent, and a CF Worker fallback without the LLM bill scaling with the number of clients.