QA Verification Contract

This page captures the three load-bearing ideas behind CareerVector's testing posture: how human-eyeballed verification is required, how the Rust quality runner consumes and emits artifacts, and what the browser end-to-end harness is allowed to spend.

Browser verification

Type checks and unit tests verify code correctness, not feature correctness. A change to anything that renders in a browser — Svelte hydration, realtime listeners, layout, modals, CV/CL previews, anything that touches window/document — is not done until it has been observed working in a real browser.

The contract:

  • The author starts a local dev server (or a Miniflare-driven workspace) and opens the affected screen.
  • The author exercises the golden path and at least one edge case.
  • If the change crosses perspectives (admin/wiki/mcp), each affected perspective is opened.
  • If the author cannot run a browser session, the work is reported as unverified, never as done.

Deployed-target browser checks count too, but they are bound by the [[cloudflare-free-tier-limits]] policy: no push-triggered e2e against a deployed Worker, no looping smoke that burns 10k requests per run, and no new scheduled cron without budget math.

The qa-platform study, the staging runbook, the live-journey runbook, the testing-architecture runbook, and the missing-journeys working memory all build on this same expectation; this doc is the canonical home.

Quality runner

The quality runner is the Rust core at tools/quality/ plus its thin shim that turns test output into release-gate signal. Its responsibilities are narrow on purpose:

  • Read run results in a stable format (JUnit-XML, machine-readable JSON).
  • Apply policy: required suites, allowed waivers, freshness windows.
  • Emit a single verdict — green / amber / red — plus a structured artifact that downstream gates can quote.
  • Persist the verdict to the quality ledger so historical trend questions ("how often does suite X flake?") are answerable without re-running CI.

The runner is not a CI orchestrator. CircleCI remains the CI runtime; the quality runner consumes its output. Replacing the CI runtime is out of scope; replacing the way we score and remember its output is the entire point.

UI E2E harness policy

Browser end-to-end coverage uses Playwright, driven against wrangler dev / Miniflare for the default lane. The policy that makes the harness cheap enough to actually run:

  • Local-first. Default lane runs against the locally booted Worker; zero Cloudflare quota consumed.
  • Deployed-target lane is manual or scheduled small. A CircleCI only_job dispatch hits the production-shaped target on demand from resource_class: corbet/unraid; a scheduled canary fires at most a handful of times per day with a tiny probe set.
  • No push-triggered deployed e2e. Past incident: 240k requests in a day. The workflow stays deleted. See wiki/content/architecture/CF-WORKERS-LIMITS.md for the post-mortem framing.
  • Helpers and fixtures live in ui/e2e/. Tests that diverge from the helpers' contract should fix the helpers first, not paper over the difference.
  • Synthetic workspace lifecycle is reset per run. Fixture data is recreated on each run so passing locally implies passing fresh, not passing against a stale browser profile.

The harness is the visible promise behind [[browser-verification]]: the machine can do most of the eye-balling, but the developer still has to be willing to open the browser when machine coverage is missing.

Source: wiki/content/runbooks/QA-VERIFICATION-CONTRACT.md