QA Platform Product Vision

Status: product vision plus first local MVP slice Date: 2026-05-19

Implementation note 2026-05-19

First API slice now exists in the shared QA platform library:

• GET /v1/runner/policy reports runner-only mode, the required corbet/unraid resource class, and the dispatchable CI job allowlist.
• GET /v1/runner/status is the dispatch gate source of truth. It combines runner inventory, running task count, unclaimed task count, available local capacity, blocked reasons, and cloud_fallback: false.
• GET /v1/behaviours, GET /v1/behaviours/:id, and GET /v1/behaviours/:id/proof expose the semantic behaviour index.
• POST /v1/qa/semantic-index accepts a generated wiki/storySpec index and writes behaviours, proof suites, suite-to-behaviour links, and suite-to-test links into D1.
• GET /v1/suites, GET /v1/suites/:id, POST /v1/suites/:id/plan, and POST /v1/suites/:id/dispatch expose test batteries as API resources.
• qa/api/migrations/0002_qa_semantics.sql adds behaviours, behaviours_fts, suites, suite_behaviours, and suite_tests.
• qa/api/src/worker.ts seeds CareerVector suite definitions from the current CircleCI lanes, keeping catalog-selected Playwright/Vitest suites visible but modeled as suite witnesses rather than a primary Tests-page dispatch surface.
• scripts/ci/build-behaviour-index.ts builds the reindex payload from tmp/story-coverage.json and, when present, tmp/test-catalog.json. Against the current repo snapshot it produces behaviours, proof suites, suite-to-behaviour links, and suite-to-test links.
• qa/ui now projects the five MVP tabs in order: Behaviour, Investigate, Queue, Suites, Tests. The header right slot shows the runner gate from /v1/runner/status, including blocked reasons and local capacity. The tabs consume the graph-shaped API surfaces rather than direct test-to-behaviour shortcuts.
• qa/ui now has behaviour detail pages at /behaviour/:behaviourId so a human can inspect the behaviour shape and suite-mediated proof tests without using MCP.
• qa/ui now gives the Investigate tab a deterministic pre-AI loop over POST /v1/investigate/changes: paste changed paths or a unified diff, get candidate tests plus suite-mediated proof paths, matched workspaces, path tokens, and skipped paths.
• scripts/ci/list-test-catalog.ts now preserves e2e compartments as catalog tags instead of leaving that knowledge only in the filesystem. The generated catalog marks Playwright rows with e2e plus layer:l1 stories, layer:l2 multi-actor, layer:l3 persistence, layer:l4 MCP contracts, layer:l5 perf, layer:l6 visual, and layer:l7 journeys where the file path proves it. The Vitest JSON parser also skips structured log lines before the actual list output, so noisy workspaces still contribute catalog rows.
• GET /v1/tests/facets summarizes the catalog by workspace, suite, status, tag, and file compartment. The Tests tab uses that API for quick facet chips, facet chips use exact tag= filters, and status=never-run maps to catalog rows whose last_status is null.
• Suite dispatch, single-job dispatch, and selected-test dispatch now call the runner gate before creating dispatch records or calling CircleCI. If the runner is not ready, the API returns 503 runner_not_ready with the full runner status payload.
• GET /v1/queue now reconciles accepted dispatch records against CircleCI before responding. The worker reads the pipeline's workflows/jobs, upserts workflow_runs and workflow_jobs, and marks the dispatch completed or failed once CircleCI reaches a terminal status. This turns manual runner dispatches into the same durable evidence ledger as ingested CI summaries.
• Manual, Suite, and Tests dispatch surfaces now show the accepted pipeline and poll queue/dispatch evidence until the matching evidence link is available.
• GET /v1/queue/dispatches/:dispatch_id exposes one dispatch audit row and reconciles it before returning when CircleCI data is available. This gives agents a narrow follow-up path after dispatch instead of asking them to scan recent history.
• The UI dispatch evidence strip now uses /queue/evidence?ids=..., a thin SvelteKit route over the targeted /v1/queue/dispatches/:dispatch_id API. Recent dispatch polling belongs to /v1/queue/dispatches.
• The QA MCP now wraps the new API resources through thin tools: get_runner_policy, get_runner_status, list_behaviours, inspect_behaviour, list_suites, inspect_suite, plan_suite, dispatch_suite, list_tests, inspect_test, investigate_changes, get_qa_graph, get_qa_neighborhood, get_ci_providers, list_queue, inspect_queue_item, list_queue_dispatches, inspect_queue_dispatch, and get_test_facets. list_tests forwards full-text q, exact tag, suite, and limit filters. Queue is the canonical execution surface; dispatch_job targets /v1/queue/dispatch.

Validation run for this slice:

bun --filter @cv/health check
bun --filter @cv/health test
bun --filter @cv/qa-platform check
bun --filter @cv/qa-platform test
bun --filter @cv/qa-api check
bun --filter @cv/qa-api test
bun --filter @cv/qa-ui check
bun --filter @cv/qa-ui test
bun --filter @cv/qa-ui build
bun --filter @cv/qa-mcp check
bun --filter @cv/qa-mcp test
bun --filter @cv/scripts-ci check
bun --filter @cv/scripts-ci test
bun scripts/audit-story-coverage.ts --json --allow-known-product-bugs
bun run scripts/ci/list-test-catalog.ts --out=tmp/test-catalog.json
bun --filter @cv/scripts-ci build:behaviour-index -- --coverage=tmp/story-coverage.json --catalog=tmp/test-catalog.json --out=tmp/behaviour-index.json

Local browser smoke:

• local qa-api: http://127.0.0.1:8790
• local qa-ui: http://127.0.0.1:5191/ or the next Vite port if occupied
• qa-ui override: QA_API_FALLBACK_BASE=http://127.0.0.1:8790 bun --filter @cv/qa-ui dev -- --host 127.0.0.1 --port 5191
• local D1 migrated through .wrangler-shared/state with migrations 0001_init.sql and 0002_qa_semantics.sql
• posted tmp/test-catalog.json through /v1/ingest/run as a catalog-only local seed: 3,172 catalog rows
• posted tmp/behaviour-index.json to /v1/qa/semantic-index: behaviours, proof suites, suite-behaviour links, and suite-test links
• checked /behaviour, /investigate, /queue, /suites, /tests
• checked /behaviour/E7
• verified header tab order: Behaviour | Investigate | Queue | Suites | Tests
• verified runner light renders in the header right slot
• verified runner gate stays visible and expandable at 1440px and 390px widths
• verified local runner status reports runner blocked, circleci_token_unconfigured, and cloud_fallback:false when no CircleCI token is present
• verified local suite dispatch returns 503 runner_not_ready before CircleCI dispatch when the runner gate is blocked
• verified Behaviour renders seeded rows from the local API
• verified behaviour detail renders 3 linked tests and links into the Tests tab
• verified Queue renders the local catalog seed item and its detail route
• verified Suites renders the configured test batteries from the local API
• verified Tests renders generated catalog rows from the local API
• verified the five tabs render without visible API error states
• screenshot: tmp/qa-ui-behaviour-local.png
• screenshot: tmp/qa-ui-behaviour-detail-local.png
• screenshot: tmp/qa-ui-tests-local.png
• screenshot: tmp/qa-ui-runner-blocked-local.png
• screenshot: tmp/qa-ui-runner-blocked-mobile-local.png

Remaining before first deployed MVP:

• deploy/apply the D1 migration in the QA API environment;
• POST the generated test catalog through /v1/ingest/run;
• POST tmp/behaviour-index.json to /v1/qa/semantic-index;
• smoke the live API endpoints after migration;
• browser-smoke the deployed UI.

North star

The QA platform is the agent-facing quality cockpit for CareerVector.

Its primary user is a coding agent trying to answer: "What behavior am I changing, which tests prove that behavior, what is the cheapest useful proof I can run now, and what evidence did the runner return?"

The build order is:

1. API first: Hono endpoints and D1 data model are the product contract.
2. MCP second: thin tools over the same API, with no separate business logic.
3. UI third: human dashboard over the API, using the shared header/footer frame.

The UI has five tabs, in this order:

1. Behaviour
2. Investigate
3. Queue
4. Suites
5. Tests

The API must make those tabs obvious without coupling itself to tab names. Each tab should be a view over durable QA resources, not a bespoke UI feature.

Hard constraints

• QA-triggered test execution runs only on the approved self-hosted runner class. The current CI adapter is CircleCI and the current runner class is corbet/unraid.
• Runner offline means QA execution offline. Do not silently fall back to cloud CI resources.
• The default spend posture is zero budget: no paid providers, quota-sensitive APIs, or live external services unless an explicit opt-in exists outside the default path.
• The product already has many tests. The platform should index and route the existing suite before inventing new test conventions.
• Behaviour is currently strongest in wiki story catalogs and weaker in code. The platform should turn that into a first-class searchable model.

Current repo assets

The product is not starting from zero.

Behaviour source

The canonical behaviour material already lives in:

• wiki/content/stories/USER-STORIES.md
• wiki/content/stories/SYSTEM-STORIES.md
• wiki/content/stories/EFFICIENCY-STORIES.md
• wiki/content/stories/story-coverage-waivers.md
• wiki/content/stories/MISSING-TESTS-HANDOFF.md

These files contain the human-readable behaviour catalog, severity, status, known debt, and the zero-budget policy.

Executable story metadata

ui/e2e/helpers/stories/story.ts already provides a lightweight BDD layer:

• storySpec({ id, persona, goal, given, when, then })
• canonical story IDs such as P1-G6-S1, X-G6-S4, ADV-S7, REC-S1, RAD-S1, S1, and E1
• Playwright annotations for story-id, persona, goal, given, when, and then[n]

This is the bridge between Behaviour and Tests.

Test taxonomy

wiki/content/runbooks/TESTING-ARCHITECTURE.md defines the semantic layers:

The important principle is that tests are organized by what they prove, not which implementation file they touch.

CI and catalog ingestion

.circleci/config.yml already has useful QA lanes:

• fast
• runner_smoke
• browser_qa
• browser_cutover
• browser_stories
• browser_contracts
• browser_dashboard
• browser_cvl
• browser_realtime
• browser_smoke
• playwright_specs
• vitest_specs
• perf
• summarize_and_ingest

All current jobs declare resource_class: corbet/unraid, and the config has an inline policy forbidding cloud fallback without an explicit structural reason.

scripts/ci/list-test-catalog.ts already lists Playwright and Vitest tests without running them, then emits a catalog that the QA API can ingest.

Existing QA API

The shared Hono app in lib/qa-platform/src/api-core/app.ts already exposes:

• health
• runner health/capacity
• runs
• gates
• tests
• artifacts
• ingest
• dispatch jobs
• dispatch suites

The existing D1 schema in qa/api/migrations/0001_init.sql has the foundation:

• workflow_runs
• workflow_jobs
• quality_events
• test_catalog
• test_results
• artifacts
• trigger_requests

The missing piece is not "a QA API". The missing piece is a semantic QA model that links behaviours, suites, test catalog rows, runner policy, and run evidence.

Research notes

BDD shape

Gherkin/Cucumber is the common external model for BDD: Feature, Scenario, Given, When, Then, tags, and domain-language examples. The relevant lesson is not that CareerVector should add Cucumber. The lesson is that behaviour should stay in the language users and domain experts use.

CareerVector's storySpec() already captures the parts we need: Given/When/Then text, story IDs, persona, goal, and a normal TypeScript/Playwright body. Adding Cucumber now would create a parallel runtime and another translation layer.

Decision: keep storySpec() as the executable BDD bridge.

Playwright metadata

Playwright supports tags, annotations, runtime annotations, report visibility, and --grep filtering. This aligns with the current storySpec() approach: keep behaviour metadata attached to normal tests, then let the QA platform index it.

Decision: use Playwright annotations and catalog output as first-class metadata. Do not introduce a second test-management metadata format for MVP.

Vitest metadata

Vitest supports listing and JSON-style output/reporters. The repo already uses vitest list --json in scripts/ci/list-test-catalog.ts.

Decision: keep Vitest as catalog-driven file/title inventory for MVP. Add richer metadata only where a real API selection need appears.

CI provider and runner policy

The current adapter is CircleCI, which uses resource_class to bind jobs to self-hosted runners. Pipeline parameters can select which workflow path runs when triggering via API. CircleCI's runner API also exposes runner inventory plus running and unclaimed task counts per resource class.

Decision: the QA API should never accept arbitrary provider commands. It should accept behaviour IDs, suite IDs, test IDs, or an allowlisted CI job for the MVP, then map them to provider parameters whose jobs are known to run on the approved self-hosted runner class. Before dispatch, the API must prove the runner is reachable, has no unclaimed backlog, has available local capacity, and that cloud fallback is disabled. CircleCI is an adapter behind /v1/ci/providers, not the QA domain model.

D1 search

Cloudflare D1 supports SQLite semantics and the FTS5 extension for full-text search.

Decision: use D1 as the MVP search engine. The corpus is small enough that external search is unnecessary. Add FTS tables for behaviours, tests, suites, and run failure text when the schema grows.

Reporting tools

Allure and ReportPortal are useful references for history, retries, metadata, attachments, and flaky-test views. They are not necessary dependencies for the MVP because the repo already ingests CircleCI summaries, Playwright artifacts, test results, and catalog rows.

Decision: borrow the product concepts, not the stack.

References:

• Playwright annotations and tags: https://playwright.dev/docs/test-annotations
• Cucumber Gherkin reference: https://cucumber.io/docs/gherkin/reference
• CircleCI self-hosted runner overview: https://circleci.com/docs/guides/execution-runner/runner-overview/
• CircleCI runner API: https://circleci.com/docs/guides/execution-runner/runner-api/
• CircleCI resource class overview: https://circleci.com/docs/guides/execution-managed/resource-class-overview/
• CircleCI pipeline parameters: https://circleci.com/docs/pipeline-variables/
• Vitest CLI: https://vitest.dev/guide/cli
• Cloudflare D1 SQL statements and FTS5: https://developers.cloudflare.com/d1/sql-api/sql-statements/
• Allure history and retries: https://allurereport.org/docs/history-and-retries/
• Allure Playwright metadata: https://allurereport.org/docs/playwright/
• ReportPortal Playwright integration: https://reportportal.io/docs/log-data-in-reportportal/test-framework-integration/JavaScript/Playwright

Domain model

Behaviour

A behaviour is a human-readable product promise.

Source examples:

• P1-G6-S1: Solo applicant tailors a CV for the first time on a job.
• S5: Quality service is a product surface.
• E2: Fast checks stay fast enough for local loops.
• RAD-S1: Future RADAR story.

Fields:

Test

A test is an executable proof candidate.

Fields extend the current test_catalog row:

Suite

A suite is a named battery of tests with intent and cost posture.

Fields:

Initial suites should mirror existing CI lanes before adding clever selection:

Run

A run is a concrete execution and its evidence.

Fields already mostly exist in workflow_runs, workflow_jobs, test_results, quality_events, and artifacts.

The API should normalize all run DTOs to the public schema at the edge. The UI and MCP should never need to know storage column names like started_at_ms or job_name.

Plan

A plan is a proposed execution set before dispatch.

The agent asks:

• "Given these changed paths, what should I run?"
• "Given this behaviour ID, what proves it?"
• "Given this failure text, what should I rerun?"

The API answers:

• selected suites
• proof witnesses
• estimated cost/runtime
• runner requirement
• rationale
• skipped or blocked work

Plans can be transient in MVP. Persist them later if they become useful audit artifacts.

API shape

The API should be boring and explicit.

Behaviour

Tests

Suites

Queue

CI providers

Planning

Runner

Data model additions

MVP additions to the existing D1 schema:

CREATE TABLE behaviours (
  behaviour_id TEXT PRIMARY KEY,
  source TEXT NOT NULL,
  title TEXT NOT NULL,
  persona TEXT,
  goal TEXT,
  severity TEXT,
  status TEXT NOT NULL,
  given_text TEXT,
  when_text TEXT,
  then_json TEXT,
  source_path TEXT NOT NULL,
  search_text TEXT NOT NULL,
  updated_at_ms INTEGER NOT NULL
);

CREATE VIRTUAL TABLE behaviours_fts USING fts5(
  behaviour_id UNINDEXED,
  title,
  persona,
  goal,
  given_text,
  when_text,
  then_text,
  search_text
);

CREATE TABLE suites (
  suite_id TEXT PRIMARY KEY,
  label TEXT NOT NULL,
  intent TEXT NOT NULL,
  budget TEXT NOT NULL,
  selection_json TEXT NOT NULL,
  circleci_job TEXT,
  runner_required INTEGER NOT NULL DEFAULT 0,
  enabled INTEGER NOT NULL DEFAULT 0,
  updated_at_ms INTEGER NOT NULL
);

CREATE TABLE suite_behaviours (
  suite_id TEXT NOT NULL,
  behaviour_id TEXT NOT NULL,
  source TEXT NOT NULL,
  updated_at_ms INTEGER NOT NULL,
  PRIMARY KEY (suite_id, behaviour_id),
  FOREIGN KEY (suite_id) REFERENCES suites(suite_id),
  FOREIGN KEY (behaviour_id) REFERENCES behaviours(behaviour_id)
);

CREATE TABLE suite_tests (
  suite_id TEXT NOT NULL,
  test_id TEXT NOT NULL,
  source TEXT NOT NULL,
  updated_at_ms INTEGER NOT NULL,
  PRIMARY KEY (suite_id, test_id),
  FOREIGN KEY (suite_id) REFERENCES suites(suite_id),
  FOREIGN KEY (test_id) REFERENCES test_catalog(test_id)
);

CREATE TABLE suites (
  suite_id TEXT PRIMARY KEY,
  label TEXT NOT NULL,
  intent TEXT NOT NULL,
  budget TEXT NOT NULL,
  selection_json TEXT NOT NULL,
  circleci_job TEXT,
  runner_required INTEGER NOT NULL DEFAULT 1,
  enabled INTEGER NOT NULL DEFAULT 1,
  updated_at_ms INTEGER NOT NULL
);

CREATE TABLE suite_tests (
  suite_id TEXT NOT NULL,
  test_id TEXT NOT NULL,
  source TEXT NOT NULL,
  updated_at_ms INTEGER NOT NULL,
  PRIMARY KEY (suite_id, test_id),
  FOREIGN KEY (suite_id) REFERENCES suites(suite_id),
  FOREIGN KEY (test_id) REFERENCES test_catalog(test_id)
);

Later additions:

• plans
• plan_items
• run_behaviours
• test_search_fts
• run_failure_fts
• runner_policy_audits

Dispatch policy

Dispatch must be intentionally narrow.

The API accepts:

• suite IDs
• test IDs
• behaviour IDs
• diff paths
• failure text

The API does not accept:

• arbitrary shell commands
• arbitrary CI job names outside an allowlist
• arbitrary resource_class
• cloud fallback flags

Implementation rules:

1. Resolve the request to an allowlisted suite/test plan.
2. Assert the target CI job is configured for the approved runner class.
3. Assert /v1/runner/status is dispatch-ready for corbet/unraid.
4. Trigger the configured CI provider with declared adapter parameters only.
5. Record a trigger_requests row before the external API call.
6. Mark dispatch accepted/rejected with provider pipeline metadata.
7. Ingest result evidence through the existing ingest path.

Add a CI/API policy check that parses .circleci/config.yml and fails if any QA-dispatchable job lacks resource_class: corbet/unraid.

UI projection

The UI should use the shared HeaderBar and footer from layout. Page files remain content-only.

The upper-right header area should show a runner light:

Tabs:

Behaviour

Human-first search over wiki stories and executable story metadata.

Primary affordances:

• full-text search
• severity/status filters
• coverage status
• linked tests
• latest proving run
• "plan tests for this behaviour"

Investigate

Future joker tab for integrated AI.

MVP should reserve the slot but avoid pretending the hard feature exists. It can show previous investigation plans later, but initially it should point at the API concept: problem text -> plan -> dispatch -> evidence.

Queue

Operational truth.

Primary affordances:

• recent CI/runner runs
• status, branch, commit, duration
• jobs, artifacts, traces
• failed tests and failure text
• linked behaviours and suites when known

Suites

Batteries of tests.

Primary affordances:

• suite intent
• runtime estimate
• budget/spend posture
• runner requirement
• dry-run plan
• dispatch

Tests

Executable library.

Primary affordances:

• full-text search by title/file/tags/story ID
• layer, workspace, suite filters
• history/flakiness
• linked suites and behaviours
• direct dispatch

MVP sequence

Phase 0: stabilize current QA surface

• Normalize API DTOs so UI and MCP consume the same shapes.
• Fix existing route drift between API, UI, and MCP.
• Keep auth out of scope for now.

Phase 1: index behaviours

• Parse wiki story catalogs into behaviours.
• Parse suite-owned selections into suite_behaviours and suite_tests.
• Add /v1/behaviours and /v1/behaviours/:id/proof.
• Add D1 FTS for behaviour search.

Phase 2: model suites

• Seed suites from existing CircleCI lanes.
• Add /v1/suites and /v1/suites/:id/proof.
• Add suite dry-run planning.
• Add runner-policy validation for all dispatchable jobs.

Phase 3: planning API

• Add /v1/investigate/changes.
• Support inputs: paths, behaviour IDs, test IDs, suite IDs, free text.
• Return suites and proof witnesses with rationale and runtime estimate.

Phase 4: UI projection

• Replace current QA UI pages with the five-tab frame: Behaviour, Investigate, Queue, Suites, Tests.
• Add runner light to the upper right of the header.
• Keep page-level chrome out of individual pages.

Phase 5: MCP thin wrapper

• Expose MCP tools that map one-to-one to API operations: list_behaviours, inspect_behaviour, list_suites, inspect_suite, list_tests, inspect_test, investigate_changes, get_ci_providers, list_queue, inspect_queue_item, list_queue_dispatches, inspect_queue_dispatch, dispatch_suite, dispatch_job, get_runner_status, and get_runner_policy.
• MCP should not duplicate selection, CI-adapter, or runner-policy logic.

Open decisions

• Behaviour parser: parse markdown tables and story bodies directly, or add a generated JSON artifact committed from a script?
• StorySpec extraction: enhance list-test-catalog.ts to preserve annotations, or add a separate Playwright metadata extraction step?
• Suite ownership: seed suites in SQL migrations, static TypeScript config, or wiki-backed YAML/JSON?
• Runtime estimates: derive from test_results only, or allow manual estimates until enough history exists?
• Investigate tab: visible placeholder in MVP, or hide until the planning API is useful enough to support it?

Working principle

The platform should not ask an agent to remember test folklore.

The agent should ask the API:

1. What behaviour is this?
2. What proves it?
3. What is the cheapest useful proof now?
4. Can the Unraid runner run it?
5. What happened?

Everything else is an implementation detail.