QA Platform Design

The architectural spec for the QA platform at qa/{ui,api,mcp}/ (CareerVector) and jobcache/qa/{ui,api,mcp}/ (jobcache). Operator dashboard, machine API, and AI-agent MCP surface for orchestrating test execution against the self-hosted CI runner.

This document is the contract. Do not deviate without updating this file first.

References: ARCHITECTURE.md (parent CV architecture), PRINCIPLES.md #1 (compute economics), #22 (CF quota), CLAUDE.md §22 (CI policy). Related source: ~/.agent/topics/server/k3s-ci-runner.md (runner deploy).

0. Scope

In scope:

• The three QA perspectives (web, api, mcp), their separation of concerns, their interconnection.
• The API as the spine — only writer to D1/R2, only caller of CircleCI, only consumer of Axiom.
• Self-hosted runner enforcement — default to corbet/unraid; cloud requires a structural reason declared inline in CI config.
• Per-spec dispatch as the operator's primary surface (not per-job).
• Telemetry: CircleCI → Axiom OTel; correlation through workflow_id.
• Multi-product sharing via lib/qa-platform/.

Out of scope:

• Test authoring (tests live in code, not in the platform). The platform discovers, dispatches, and reports.
• General observability for production CV/jobcache apps (separate datasets, separate config).
• Release management (no tags; direct-to-main per project memory).

1. Mental model

Operator ──┬─→ qa-ui (UI, SvelteKit on CF Workers)         ──┐
           ├─→ qa-mcp (MCP server, CF Workers)              ──┼─→ qa-api (Hono on CF Workers)
           └─→ curl / scripts                              ──┘     │
                                                                   │
                                                ┌──────────────────┼──────────────────┐
                                                ▼                  ▼                  ▼
                                            D1 (data)         R2 (blobs)         CircleCI
                                                                                      │
                                                                                      ▼
                                                                            corbet/unraid k3s
                                                                                      │
                                                                                      ▼
                                                                          OTel → Axiom (EU)

The api is the only writer to D1 and R2, the only caller of CircleCI's v2 API, the only consumer of Axiom for trace deeplinks. UI/MCP/scripts are pure consumers of the api. CI ingest is the only push-side caller.

2. Domain entities

Schemas live in @cv/health (lib/health/src/*.ts). The platform extends but doesn't fork that package.

3. API surface

/v1/* prefix from day one. Auth is deliberately stripped for the current private-admin sprint; the recovery source lives in wiki/content/archive/2026-05/auth-stashed/. Paths below reflect the shipped surface in lib/qa-platform/src/api-core/routes/.

3.1 Catalog

GET    /v1/tests                     filters: workspace, suite, file_glob, status, limit, since
GET    /v1/tests/:id/proof           one witness: suites, behaviours, implementations, graph validity
GET    /v1/tests/:id/history         per-test run history with limit
GET    /v1/tests/:id/flake           flake stats over 7d/30d/90d windows
GET    /v1/behaviours                Gherkin-shaped behaviour index
GET    /v1/behaviours/:id/proof      one behaviour: idea doors, suites, test witnesses
GET    /v1/suites                    proof-set catalog
GET    /v1/suites/:id/proof          one proof set: behaviours, test witnesses, graph validity
POST   /v1/investigate/changes       change-to-proof graph investigation
                                     body: { paths?: string[], diff?: string, branch?: string }
                                     → { proofs: TestProof[], counts, matched_* }

Test and suite proof payloads include a shared QA graph slice:

{
  graph: {
    center: string,
    nodes: Array<{ id: string, kind: string, label: string }>,
    edges: Array<{ from: string, relation: string, to: string, source?: string | null }>,
    validity: LocalValidityResult[],
    gaps: { implementation?: boolean, suites?: boolean, behaviours?: boolean, tests?: boolean }
  }
}

The canonical proof path is idea ← behaviour ← suite ← test → implementation. A green gaps object only means the required link categories are present; validity still reports local graph requirements such as a suite being too small.

3.2 Queue

GET    /v1/queue                     filters: branch, lane, status, limit, since
                                     → Future/Now/Past queue rows + summary counters
GET    /v1/queue/:workflow_id        detail (run + jobs + quality_events)
POST   /v1/queue/:workflow_id/refresh
                                     refresh CircleCI workflow/job state, artifacts, test metadata
POST   /v1/queue/:workflow_id/cancel cancel a CircleCI workflow, then best-effort refresh
POST   /v1/queue/:workflow_id/rerun  body: { from_failed?, jobs?, enable_ssh?, sparse_tree? }
                                     → new CircleCI workflow_id
POST   /v1/queue/reorder             body: { queue_ids: string[] }
                                     persist future dispatch order only
GET    /v1/queue/:workflow_id/trace  { axiom_deeplink, trace_id, dataset, apl_query }
GET    /v1/artifacts/:run_id/:name   302 redirect to a signed URL (15-min TTL)
GET    /v1/artifacts/raw/*           internal proxy backing signed-URL redirects

Queue index is the compact operational timeline. The API merges dispatch audit rows from trigger_requests with observed CircleCI workflow evidence from D1, then projects every row into one of three lanes:

• future: requested/accepted dispatch rows without workflow evidence; reorderable.
• now: running or accepted work that already has workflow evidence; never reorderable.
• past: completed or failed workflow/dispatch evidence; never reorderable.

Open Queue evidence ages out after six hours. A stale requested/accepted dispatch or pending/running/unknown workflow is projected into past with phase: "stale" and status: "unknown" so "Now" remains an active-work lane instead of an archive of old importer gaps.

Queue detail is now the operational CircleCI evidence surface. GET /v1/queue/:workflow_id performs a best-effort live refresh when CircleCI is configured, then returns cached D1 state plus a circleci summary (refreshed_at, artifact count, test row count, non-fatal evidence errors). Per-job rows include CircleCI job ids/numbers, artifact links, and summarized test counts when available. The POST action routes are the API-first contract used by MCP and the UI; they never bypass qa-api to call CircleCI directly. POST /v1/queue/reorder is deliberately narrow: it updates only requested/accepted dispatch rows where workflow_id IS NULL, so historical or in-progress evidence cannot be rewritten by dragging the UI or by an MCP tool call.

3.3 Dispatch

Two flavors shipped: named job dispatch and suite dispatch. Tests are witnesses inside suites, not a primary dispatch object. Dispatch writes a D1 audit row, checks runner capacity before burning CI time, and POSTs to CircleCI's pipeline-definition /run endpoint.

GET    /v1/queue/jobs                static allowlist (named jobs runnable from here)
GET    /v1/queue/dispatches          recent dispatch audit rows (Queue evidence)
GET    /v1/queue/dispatches/:id      one dispatch audit row + reconciled workflow evidence
POST   /v1/queue/dispatch            body: { job, branch, parameters?, reason, requested_by? }
                                     → named-job dispatch (browser_qa, perf, runner_smoke, ...)

Dispatch response shape:

{ "dispatch_id": "...", "pipeline_id": "...", "pipeline_number": 0, "pipeline_url": "...", "dispatched_at": "..." }

No separate sweep endpoint: POST /v1/queue/dispatch with the appropriate job name covers full-config sweeps.

3.4 Graph

GET    /v1/graph/qa                  product QA graph projection
GET    /v1/graph/qa/neighborhood     center + max_depth + direction + optional relations/kinds

The projection is the same adapter used by test proof, suite proof, Investigate, MCP tools, and the Wiki Atlas links. The route is read-only and graph-shaped; it does not write annotations or touch Falkor.

3.5 Gates

GET    /v1/gates                     per-gate rollup over a default window
GET    /v1/gates/:gate/history       paged event list for one gate

3.6 Runner health

GET    /v1/runner/health             { reachable, ready, last_heartbeat, k3s_version }
GET    /v1/runner/active             last-used / version metadata (no live pod listing)
GET    /v1/runner/capacity           { max_concurrent, in_use, queued }

Queue dispatch consults /v1/runner/capacity before dispatch. If full → 503 with Retry-After.

3.7 Ingest (CI → api)

POST   /v1/ingest/run                summarize_and_ingest payload (CIWorkflowSummary + results + catalog)

Catalog rows ride in on the same payload — there is no separate /v1/ingest/catalog endpoint.

3.8 Liveness

GET    /v1/healthz                   { ok, version, dbMs } — no auth

4. Cross-cutting concerns

4.1 Auth

Auth is an outer admin boundary, not a per-route concern inside qa-api or qa-mcp. Once a caller reaches these surfaces, treat them as a trusted DMZ: typed operations, validation, idempotency, and audit rows still matter, but do not rebuild bearer/scope checks incidentally inside each route. If the outer gate changes, it should land as a deliberate slice using the archived recovery source.

4.2 Idempotency

• All POSTs accept Idempotency-Key header.
• qa-api stores (idempotency_key, response_json, expires_at) for 24h.
• Repeat with same key returns the cached response; doesn't re-dispatch.

4.3 Rate limiting / backpressure

• Queue dispatch reads /v1/runner/capacity before creating CircleCI work. If in_use >= max_concurrent, returns 503 with Retry-After: N.
• No global rate limit needed — admin-only surface, low traffic.

4.4 Self-hosted enforcement

• Dispatch calls CircleCI's /api/v2/project/{slug}/pipeline/run with definition_id (per-product Pipeline Definition UUID, configured as a wrangler secret) and parameters shaped for the workflow. Auth uses a GitHub-App-scoped CircleCI token, not a personal token.
• The CircleCI config (.circleci/config.yml) defaults every CV job to resource_class: corbet/unraid. No job currently runs on cloud resources.
• The CircleCI runner registered for that resource class is named corbet-unraid-runner; the k3s Helm release remains container-agent.
• Structural exception is the only escape: an inline CLOUD-REQUIRED: comment with a written justification, plus the cloud resource_class. Today no CV job uses this hatch.
• The api has no concept of "cloud dispatch" — CircleCI routes per-job from the config.

4.5 OTel correlation

• CircleCI's OpenTelemetry export ships pipeline traces directly to Axiom over HTTP (https://eu-central-1.aws.edge.axiom.co/v1/traces, header Authorization: Bearer xaat-…, X-Axiom-Dataset: cv_telemetry_dev). Configured in the CircleCI org dashboard, not in code. The gRPC exporter was tried and removed — HTTP is the only working path.
• Each pipeline trace carries circleci.workflow_id as a resource attribute.
• trace_id = workflow_id with hyphens stripped and lowercased (32-char hex, OTel-compliant) — derived in queue.ts, not stored.
• /v1/queue/:workflow_id/trace returns an Axiom deeplink shaped:

  https://app.axiom.co/{org}/trace?traceId={hex}&traceDataset={dataset}
  

  plus the raw APL query so an operator can paste it into Axiom directly. If AXIOM_ORG_SLUG is unset the deeplink is omitted; the response still surfaces trace_id + apl_query.
• qa-api itself emits OTel spans for its own operations to the same dataset. Operators see "dispatch X → CircleCI pipeline Y → test results Z" as one correlated trace.

5. CI integration

5.1 Dispatch jobs

.circleci/config.yml exposes two generic jobs the api dispatches by name. Both pin resource_class: corbet/unraid:

jobs:
  playwright_specs:
    resource_class: corbet/unraid
    parameters:
      specs:
        type: string
        default: ""
      target_url:
        type: string
        default: "wrangler-dev"
    steps:
      - checkout
      - install_workspace
      - run:
          command: |
            if [ "${TARGET_URL}" = "wrangler-dev" ]; then
              # spawn local wrangler dev inside the pod
            else
              export PLAYWRIGHT_BASE_URL="${TARGET_URL}"
            fi
            bun --filter @cv/ui test:e2e -- ${SPECS}

  vitest_specs:
    resource_class: corbet/unraid
    parameters:
      specs:
        type: string
        default: ""
      workspace:
        type: string
        default: "@cv/*"
    steps:
      - checkout
      - install_workspace
      - run:
          command: bun --filter ${WORKSPACE} test ${SPECS}

Focused Queue dispatch fans suite-linked test ids into these two jobs based on suite type.

Plus the named jobs (fast, browser_*, perf, runner_smoke, summarize_and_ingest) dispatched via POST /v1/queue/dispatch. All on corbet/unraid by default.

5.2 The summarize_and_ingest job

After every workflow, summarize_and_ingest (depends on every other job) runs:

• Merges per-job artifacts (circleci-summary-*.json, test-results-*.json)
• Uploads heavy artifacts (Playwright trace zips) to R2 directly via S3-API (aws s3 cp against the R2 endpoint)
• POSTs the merged payload + catalog rows to https://api.qa.careervector.corbet.ch/v1/ingest/run with Authorization: Bearer ${QA_INGEST_BEARER} (CircleCI env var)
• Exits 0 always — never fails CI

5.3 Pipeline parameters

The main pipeline definition declares:

parameters:
  only_job:     { type: string,  default: "auto" }   # named-job filter
  bypass_fast:  { type: boolean, default: false }    # skip fast lane for targeted runs
  specs:        { type: string,  default: "" }       # spec list for playwright_specs/vitest_specs
  target_url:   { type: string,  default: "wrangler-dev" }
  workspace:    { type: string,  default: "@cv/*" }

Per-job filters: expressions gate which jobs run for given parameter values. The api sets parameters appropriately per dispatch flavor.

6. The qa platform packages

Shipped layout:

lib/qa-platform/                     # @cv/qa-platform — shared library
  src/
    api-core/
      app.ts                         # makeQaApiApp(config) → Hono app
      env.ts                         # Env contract for consumers
      index.ts                       # re-exports
      routes/
        health.ts                    # /v1/healthz
        tests.ts                     # /v1/tests, /v1/tests/:id/{history,flake}
        behaviours.ts                # /v1/behaviours, /v1/behaviours/:id/proof
        suites.ts                    # /v1/suites, /v1/suites/:id/{proof,plan,dispatch}
        investigate.ts               # /v1/investigate/changes
        graph.ts                     # /v1/graph/qa, /v1/graph/qa/neighborhood
        queue.ts                     # /v1/queue[/...], /v1/queue/:id/trace, /v1/queue/dispatches
        gates.ts                     # /v1/gates[/...]
        artifacts.ts                 # /v1/artifacts/:run_id/:name, /v1/artifacts/raw/*
        ingest.ts                    # /v1/ingest/run
        queue-dispatch.ts            # /v1/queue/{jobs,dispatch}, /v1/ci/providers
        runner.ts                    # /v1/runner/{health,active,capacity}
      middleware/{auth,logging}.ts
      lib/{circleci,flake,gateRollup,r2-sign,testId}.ts
      db/{client,queries/{tests,runs,gates,triggers,ingest}}.ts
    mcp-core/
      index.ts                       # makeQaMcpApp(config) → Hono JSON-RPC server
      qa-rest.ts                     # client wrapping the qa-api bearer flow
      types.ts
      tools/
        index.ts
        _helpers.ts
        list_queue, inspect_queue_item,
        list_tests, get_test_history, get_flakes,
        list_gates, get_gate_history,
        dispatch_suite, dispatch_job # queue dispatch tools
      middleware/                    # tool-call validation
    ui/
      index.ts
      components/                    # Svelte 5 QA dashboard components — nav, queue, tests, dispatch
        nav/TabNav.svelte
        queue/{QueueTable,StatusBadge}.svelte
        tests/{TestsCatalog,FlakeBar}.svelte
    web-server/
      api-client.ts                  # typed fetch client used by qa-ui SSR/loaders
      index.ts
    db/
      index.ts                       # init SQL schema (currently re-exported, not centralized)

qa/        api/  mcp/  ui/           # CV thin wrappers (~80 LOC of worker.ts each + wrangler.toml)
jobcache/  api/  mcp/  ui/           # jobcache thin wrappers (siblings; build green, deploy pending DNS)

Each thin wrapper passes product-specific config to the library factories:

• productName ("careervector" | "jobcache")
• circleciProjectSlug
• circleciPipelineDefinitionId (per-product UUID)
• workflowAllowlist (per-product subset of named jobs)
• axiomDataset + axiomOrgSlug
• corsAllowlist
• D1/R2 bindings + token-map secret

7. Shipped vs Open

Shipped

Open

• jobcache CF infra (Phase 2c). D1 + R2 + DNS for qa.jobcache.corbet.ch and siblings is still pending — Pulumi declarations not yet applied.
• LLM smart selection. Affected-by today is path-glob only. LLM-ranked selection is deferred until path-glob proves insufficient.
• Scheduled smoke probes from qa-api. Cloudflare Cron Trigger → runner_smoke dispatch → Axiom check. Cadence undecided.
• Sweep semantics divergence. No separate sweep endpoint shipped; revisit only if resource accounting becomes a feature.
• Test catalog refresh cadence. Today every fast run re-ingests. May move to on-demand if catalog churn matters.

8. What this is NOT

• Not a test runner. CircleCI executes; the platform orchestrates and reports.
• Not a CI replacement. CircleCI keeps its dashboard, logs, env vars, contexts. This platform sits on top.
• Not multi-tenant. Each product has its own D1, R2, secrets, DNS. No path-based or header-based tenant routing.
• Not a general observability platform. Axiom holds the production CV/jobcache app telemetry. The QA platform consumes Axiom for trace deeplinks only.
• Not a place to add ad-hoc business logic. If a feature isn't "select tests / dispatch / report results / show health", it doesn't belong here.

9. Known limitations

• Shared library tests are focused, not exhaustive. lib/qa-platform/ now has direct vitest coverage for the graph route, proof path, and MCP graph-neighborhood tool, while consumer wrapper tests still cover deployed worker wiring.
• Per-product init SQL is duplicated. Each product owns its migrations/0001_init.sql and they drift independently. The lib/qa-platform/src/db/ directory exists but doesn't yet hold a centralized schema.
• Affected-by heuristic is single-mode. Path-glob only (workspace match + path-token match). The heuristic field in the response is always 'path-glob'. No diff parser, no LLM ranker.
• /v1/runner/active doesn't list current task pods. It returns last-used / version metadata only. A real pod listing would require k8s API access from the worker — not wired.
• Dispatch responses are audit-only. No log streaming, no progress callbacks. Operators poll /v1/queue/:workflow_id or watch Axiom for live state.

10. Live URLs

CV (deployed, Pulumi-managed custom hostnames):

• qa.careervector.corbet.ch — qa-ui
• api.qa.careervector.corbet.ch — qa-api
• mcp.qa.careervector.corbet.ch — qa-mcp

jobcache (wrappers built, deploy pending Phase 2c):

• qa.jobcache.corbet.ch — qa-ui
• api.qa.jobcache.corbet.ch — qa-api
• mcp.qa.jobcache.corbet.ch — qa-mcp

Pulumi (infra/cloudflare.ts) declares custom domains; pre-Pulumi Worker Routes and CNAMEs were removed 2026-05-17.

11. Operational artifacts

12. Open decisions

• LLM smart-selection provider / model / prompt budget. Defer until path-glob is demonstrably insufficient.
• Smoke-probe cadence. Hourly? 4×/day? Needs operator-noise calibration before wiring the cron.
• Centralized init SQL. Decide between baking schema into lib/qa-platform/src/db/migration.ts (one source of truth, products diverge via overrides) vs keeping per-product migrations.