Testing Architecture

The foundation for every test in CareerVector. Read this once; write tests against the layers it defines.

Mission

Per wiki/content/architecture/PRINCIPLES.md §4: tests are the spec, not regression guards. The product's behavior is what we test; the implementation is what we refactor. A test that breaks when an internal helper is renamed is poorly written. A test that breaks when the user-visible behavior changes is doing its job.

CareerVector has shipped many architectural waves — multi-actor primitives, IDB persistence, Y.Text collaborative prose, pipeline stage extraction, structured provenance, the human-write protection gate, SSE fanout — and the existing test surface has grown unevenly under those waves. The job of this architecture is to give every future change a layer to land its tests in, so coverage stays honest as the product evolves.

Layered taxonomy

Tests are organised by what the test is proving, not by what file the implementation lives in. A "drag a job to Applied" test is a story-level spec regardless of whether the implementation is React, Svelte, an op envelope, or a hand-rolled dispatchEvent. Refactor the inside; the test stays.

Below this sit the package-internal vitest suites in packages/<name>/tests/ and apps/<name>/test/. Those are framework-agnostic, they always run, and they're where pure logic / op invariants / SDK contracts get their first proof.

The seven layers above are the new, named, semantic surface. Each layer has a directory, a helper module, a reference test, and a CI policy.

L1 — User story specs

A story spec is a single user goal, written in Given/When/Then form, using the user's vocabulary. No implementation details. No selectors in the spec body — selectors live in helpers.

Format

storySpec({
    persona: 'solo applicant',
    goal: 'track a job through pipeline stages',
    given: 'a workspace with one imported job',
    when: 'I drag the job to "Applied"',
    then: [
        'the job shows in "Applied" on this tab',
        'the job shows in "Applied" after a reload',
        'a collaborator opening the same workspace sees the job in "Applied"',
    ],
}, async ({ workspace, dashboard, secondTab }) => { ... });

The storySpec helper (see ui/e2e/helpers/stories/story.ts) wraps Playwright test() with a structured header that becomes the spec's title and prepares the shared fixtures (workspace, dashboard, secondTab, etc.).

Index

Story specs are indexed by persona × goal. Personas are the actor types we design for:

Goals are user intents, not features. "Track a job through pipeline stages" is a goal; "drag-and-drop functionality" is a feature. Test goals.

Anti-patterns

• DO NOT put data-testid strings in spec bodies. Helpers do the resolving.
• DO NOT mention Y.Doc, applyOp, batch, nanoid in spec text. Those are implementation.
• DO NOT chain unrelated assertions into one story. One goal per story.
• DO NOT rely on global state across stories. Every story creates its own workspace.

Why a separate layer

Existing e2e specs (cvl-reorder.spec.ts, journeys.spec.ts, qa-gate.spec.ts) are valuable but mix three things: smoke verification, behavior assertion, and bug-pinning regressions. Story specs separate the user contract from the smoke layer (qa-gate stays for that) and from the bug-pin layer (known-bugs/ stays for that). A story spec is the spec form of the product; if it doesn't pass, the product is broken at the user level.

L2 — Multi-actor convergence harness

V3 realtime is the product's hardest claim. Multiple actors edit the same workspace concurrently and converge on one canonical state. Today most tests are single-actor; the few multi-actor specs are mostly single-tab + single-API-write.

The convergence harness spawns N "actors" — each one a Playwright BrowserContext, an MCP HTTP client, or a server-fallback simulator — and runs an interleaved op script against them. After draining, it asserts every actor's view of the workspace matches a single canonical answer.

Actor types

Convergence test shape

multiActorTest({
    actors: ['browser-a', 'browser-b', 'mcp', 'api'],
    setup: async ({ workspace, actors }) => { ... },
    operations: [
        { actor: 'browser-a', op: { kind: 'job.update', id: 'j1', patch: { status_id: 5 } } },
        { actor: 'mcp', op: { kind: 'job.update', id: 'j1', patch: { salary: 180 } } },
        { actor: 'browser-b', op: { kind: 'selection.update', ... } },
    ],
    expect: async ({ workspace, actors }) => {
        const canonical = await readCanonicalState(workspace);
        for (const actor of Object.values(actors)) {
            const view = await actor.read();
            expect(view).toEqual(canonical);
        }
    },
});

The harness handles the ordering quirks — letting the realtime broadcast settle, draining the coalesce buffer, waiting for IDB writes — so spec authors only describe the operations and expectations.

Invariants the harness checks for every test

1. Convergence — every actor's final view equals the canonical state.
2. OpId dedup — replaying the same op (same opId) does not double-apply.
3. No lost writes — every op landed in workspace_sub_doc_ops exactly once.
4. Order independence — the same set of ops applied in any order produces the same final state (where the ops are commutative, which Yjs guarantees for most).

Why this is its own layer

These tests are inherently slow (multi-context spin-up), error-prone (many timing windows), and require dedicated infrastructure (the harness). Mixing them with L1 stories would make L1 slow. Pulling them out into their own layer means L1 can stay fast and L2 can take the time it needs.

L3 — Persistence + recovery contracts

Writes have to survive bad networks, browser crashes, tab reloads, and IDB upgrades. Each of those is a contract.

Test categories

Helper

ui/e2e/helpers/persistence/lifecycle.ts exposes killConnection(actor), simulateOffline(actor), freshContext(workspaceId), and expectConvergedAt(actor, snapshot).

Why this is its own layer

Reload survival is currently smeared across many L1 specs (cvl-reorder.spec.ts reloads, qa-gate.spec.ts reloads). When a reload-related bug appears, the search space is huge. Centralising in L3 means: one place to look, one place to add regression coverage.

L4 — MCP behavior contracts

The MCP server is its own product surface. Per PRINCIPLES.md §7 (Agent API parity), every human gesture has an MCP equivalent. Per §18 (no identity inference), every MCP tool requires explicit identifiers. These are contracts the layer enforces.

Test categories

Test client

ui/e2e/helpers/multi-actor/mcp-client.ts is a thin HTTP client that speaks the MCP JSON-RPC wire format. It does not import @modelcontextprotocol/sdk — the SDK is for production clients; tests use a hand-rolled client to keep dependencies thin and to assert on raw wire shapes.

Why this is its own layer

The MCP worker is deployed separately, has its own wrangler.toml, and runs against the ui worker via service binding. Testing it as if it were just another e2e surface would tangle dependencies. Layer 4 owns the MCP contract independently.

L5 — Performance budgets

Per PRINCIPLES.md §1 (compute economics), perf is a design concern. Budgets make it a CI gate.

Per-surface budgets

Budgets are stored in ui/e2e/perf/budgets.json so a future tightening pass is a single-file change.

How a budget fails

A perf test runs N iterations (default 5), takes the p95, and asserts p95 < budget. Failure logs the actual numbers and a 60-day rolling history (committed under ui/e2e/perf/history/).

Grace period for new tests

When a new perf test is added, its first PR sets the budget to Math.ceil(p95 * 1.10) (10% headroom over measured value). The author may relax this once for clear documented reasons. After landing, the budget can only tighten — no upward drift without a // reason: ... comment.

Local vs CI

Local runs print numbers; CI runs assert. The same suite runs in both; PERF_ENFORCE=1 switches assertion mode on.

L6 — Visual regression

Pixel-diff for layout-critical surfaces. Per feedback_look_at_images.md in the user's memory: "any visible difference is a regression."

Surfaces under visual regression

Snapshot strategy

Playwright's built-in expect(page).toHaveScreenshot(). Snapshots checked into ui/e2e/visual/__snapshots__/. Diff threshold: maxDiffPixels: 50 (allows minor anti-aliasing variation).

When a snapshot changes intentionally

bun --filter @cv/ui test:visual:update regenerates. Commit the new snapshots with a message describing the intentional change.

Why not Chromatic / Percy

Both are paid SaaS. CareerVector is on free-tier Cloudflare and a strict no-spending posture. Playwright's built-in works, snapshots live in git, and we have full control. If volume ever justifies SaaS, the migration is straightforward.

L7 — Real-world journey flows

End-to-end against real URLs, real LLM providers, real Typst rendering. Gated by env vars so dev runs stay free.

Journeys

Why gated

Per CLAUDE.md §22, CF free-tier is 100k requests/day account-wide. L7 burns real quota; running it on every PR would 1027 production. Gate it with env vars, run it manually before releases or on a nightly cron with budget math.

Local-only L7 patterns

For an wrangler dev / Miniflare run, L7 falls back to live LLM via real provider API. This is still a cost gate (LLM tokens) but does not touch CF quota.

Below the layers: package-internal vitest

Some tests don't fit the user-story shape because they're testing pure logic at the bottom of the stack: op apply, schema validation, scoring math, cell-origin projection, locale resolution.

These live in packages/<name>/tests/ and apps/<name>/test/. They're always run (no gate), they're fast (under 30s for the whole suite), and they're framework-agnostic. The existing tests in lib/domain/tests/ and lib/mutations/test/ are the model.

Three sub-categories:

Mutation testing

StrykerJS is installed in lib/mutations and lib/domain. Configs are checked in (stryker.conf.json). Run with bun --filter @cv/mutations mutate. Use for high-confidence layers — the apply path, the gate logic, the cardinality validator. Don't run on every PR (too slow); run nightly or before significant refactors.

Framework choices

Justifications

No Gherkin / Cucumber. The cost of a separate spec syntax (parsed at runtime, separate IDE support, no type checking on spec → code binding) outweighs the benefit. storySpec() gets us 90% of the readability with no runtime cost and full TypeScript.

No Playwright Component Tests. The feature is experimental and the Svelte 5 support story is still rough. When Svelte components have logic worth testing in isolation, lift the logic into lib/domain/ and unit-test it there. Visual behavior is e2e.

No Chromatic / Percy. Free-tier ethos. Playwright built-in works.

Mutation testing nightly, not per-PR. A full Stryker run can take 30+ minutes. Per-PR would slow every change. Nightly catches regressions in invariant layers and produces an HTML report.

Spec authoring vocabulary

Below is the only vocabulary spec authors should use in spec text (titles, descriptions, given/when/then). Implementation terms are forbidden; if you need to refer to one, use a helper.

Allowed (user-visible)

• Workspace, job, applicant, collaborator, agent
• Dashboard, kanban, CV editor, CL editor, preview
• Pipeline stages: extract, enrich, evaluate, tailor (these surface to the user)
• Status names ("Applied", "Interview", etc.), view names ("By Priority"), industry names
• "Drag", "click", "type", "navigate", "share link", "reload"
• "Appears", "disappears", "renders", "is visible", "is hidden"

Forbidden (implementation)

• Y.Doc, sub-doc, op, batch, opId, nanoid, IDB, IndexedDB
• Selectors, data-testid, CSS classes, role attributes
• HTTP, REST, PATCH, PUT, fetch
• Worker, DO, durable object, broker, relay
• xstate, machine, state, transition
• Yjs, CRDT, awareness, sync vector
• React, Svelte, $state, $derived, hydration
• Cloudflare, D1, edge, miniflare, wrangler

Any of these in a spec title is a code smell. Push them into helpers.

CI integration

Per CLAUDE.md §22, no push-triggered runs hit deployed Workers. All CI runs locally against wrangler dev / Miniflare. Deployed-worker runs are scheduled or manual.

Per-PR (push-triggered)

Runs locally in CI containers against Miniflare. Total budget: 15 minutes.

Nightly (scheduled, off-hours)

Bigger budget. Allowed to spend tokens.

Pre-release (manual)

Same as nightly + deployed-worker realtime tests (PLAYWRIGHT_BASE_URL=https://careervector.corbet.ch test:qa:realtime).

Workflow files

This document describes the policy. The actual workflow YAMLs are not committed (per CLAUDE.md §22, push-triggered deployed-worker runs are forbidden, and we deliberately keep workflows minimal). When a workflow is added, it must:

• Run against Miniflare for L1-L6
• Be scheduled (not push-triggered) for L7
• Stamp the configured Cloudflare quota check before any deployed-worker work

Local dev

Decisions on the existing test surface

The existing ui/e2e/*.spec.ts files are not deleted. They continue to run via bun --filter @cv/ui test:e2e. The new layers are additive. Over time, agents picking up a spec for maintenance should consider whether the spec is best expressed as L1 (move + rewrite), L2 (the multi-tab portion lifts), L3 (the reload portion lifts), or stays where it is.

The existing helpers/api.ts, helpers/cvl.ts, helpers/dnd.ts, helpers/quality.ts, helpers/settle.ts, helpers/demo.ts remain. They're working and battle-tested. The new helpers under helpers/stories/, helpers/multi-actor/, and helpers/perf/ are supersets, not replacements. New helpers compose old helpers; old helpers don't compose new ones.

Known sharp edges (diagnosis)

• cvl-reorder.spec.ts Journey 1 occasional flake — y-indexeddb write race. The 2_000ms timeout is best-effort; the real fix is asserting on IDB write completion via the SDK's subscribe() callback. L3 has the pattern.
• agent-api-parity.spec.ts:233 — surfaces a real divergence between API-tailoring and GUI-tailoring. Don't pin to test.fail — this is the kind of bug L1 stories should catch and L4 MCP contracts should encode permanently.
• canonical-collaboration.spec.ts — already a good template for L4-style API-only assertions. Keep it as-is; L4 grows around it.
• mcp-dispatch-headless.spec.ts — exemplary headless-MCP test. L4 reference.
• qa-gate.spec.ts — exemplary smoke gate. Keep using it. Don't expand it; expansion goes into L1.

Spec author quick reference

I want to test...               Layer  Where
─────────────────────────────────────────────────────
A user goal end-to-end          L1     e2e/stories/
N tabs converging               L2     e2e/multi-actor/
Reload / offline / IDB          L3     e2e/persistence/
MCP tool call → state           L4     e2e/mcp-contracts/
Latency / throughput            L5     e2e/perf/
Layout stability                L6     e2e/visual/
Real LinkedIn + LLM             L7     e2e/journey-real/
Pure op / scoring / locale      below  packages/<x>/tests/
Mutation kill rate              below  bun --filter @cv/<x> mutate

If a test crosses layers, split it. A "drag job to Applied + verify on another tab + verify on reload" test is three tests: one L1 story, one L2 convergence, one L3 persistence. Each independently fails for an independently fixable reason.

What "good" looks like

A new test agent arrives in the repo and:

1. Reads this doc (10 minutes).
2. Identifies which layer their feature belongs in (1 minute).
3. Opens the layer's reference test, copies the shape, replaces the contents (15-30 minutes).
4. Runs locally, lands the test green.
5. Test continues to pass through 6 months of unrelated refactors.

If any of those steps doesn't work, this doc has a bug. Fix it.

Maintenance

This doc is a living artifact. When a layer's reality drifts from this description, update the doc, don't paper over. The doc is the spec; tests are the spec; refactoring the docs is a real change.

Owner: whichever agent or human is doing significant test work this week. Trade off and document changes in commit messages.