Story coverage waivers

The audit script scripts/audit-story-coverage.ts separates traceability from green status:

  • GREEN means every current Critical/High story is covered without debt.
  • PASS_WITH_DEBT means every Critical/High gap is classified, but accepted debt remains. This is not green.
  • FAIL means missing coverage, broken story IDs, unclassified debt, skipped required proof, or a Critical/High known product bug.

Coverage debt is loud by design: a missing spec is a real product risk, and quiet "missing" rows lead to the same drift the audit was created to prevent.

Tone vocabulary is shared with the quality surface:

  • green — covered and current
  • yellow — accepted debt, needs-harness work, or caveated/proxy proof
  • red — product bug, broken proof, missing proof, skipped required proof, or invalid proof
  • blue — manual/live-provider-only proof
  • gray — future/deferred/out-of-current-scope proof

When a story genuinely cannot be tested yet, list it here with one of these reason categories — the same vocabulary used by // coverage-status: comments in spec files:

  • future product — the product feature isn't built yet
  • manual/live-provider only — only provable against a paid/live provider
  • needs harness — test infrastructure must land first
  • known product bug — product is misbehaving; Critical/High rows fail the default gate until fixed
  • obsolete story — catalog row is wrong / superseded; clean it up next

The audit treats waived entries as explicit coverage debt (status debt), not silent successes. They are visible in the table and counted separately. Adding to this file should always be paired with a follow-up task to make the spec real, or with a doc change to remove the catalog row. Do not use this file to hide Critical/High product bugs; those should make CI red unless the audit is being run deliberately in inventory mode with --allow-known-product-bugs.

Schema

<ID>:
  category: <one of the categories above>
  note: <short, factual reason — what unblocks coverage>

The file is read as YAML-like flat key/value blocks. Lines starting with # are comments. IDs that don't exist in any catalog are ignored.

Waivers

# System stories — quality service and provider failure matrix still in design.
# S2 is now covered by lib/domain/tests/ai.test.ts (see the "S2 —
# provider failure normalization matrix" describe block). The matrix asserts
# (provider × failure-mode) coverage, no API-key leakage in error surfaces,
# and provider-neutral envelope shape.
# S5 is now covered by tools/quality/src/http.rs — a minimal HTTP/1.1
# surface (std::net only, no framework deps; keeps the README "stdlib-only"
# promise) exposing /healthz /status /gates /buckets /runs. Run via
# `quality serve --addr 127.0.0.1:9123`. Integration tests at
# tools/quality/tests/http_smoke.rs.
#
# S6 is now covered by the same surface: every /runs request appends a
# structured event (LedgerEvent::to_json_line) carrying bucket + status +
# provider + note to the configured ledger file. Tests assert ledger
# contents after a /runs POST.

# Efficiency stories — same shape as System; depend on the same harness work.
# E2 is now covered by scripts/check-fast-suite-duration.ts. Budget seeded
# at 60 s (~4× observed local wallclock for CI headroom). Per
# PRINCIPLES.md #4, tighten over time; loosening requires a reason comment
# at the top of the script. History persisted to tmp/fast-suite-duration.json
# accumulates rolling p95 for future seed tightening.
# E3 is now covered by scripts/check-flakiness-budget.ts. The contract is:
# Playwright config carries `retries: 0`. Retries mask flake; the catalog
# claim "repeatable" only holds if a passing run is deterministically
# passing. When per-spec retry-count history accrues from CI, the sentinel
# can grow into a runtime check; for now the static contract is enforced.
# E4 and E5 are now proven by scripts/check-paid-forbidden.ts (see story-id
# markers there). Do not re-add the waivers; if you ever weaken the sentinel,
# the audit will catch it as the IDs going un-covered.
# E6 is now covered by the same HTTP surface. POST /runs with a
# paid-forbidden gate returns 200 with status="blocked" and a note
# explaining the policy refusal — exercised by the
# `run_paid_forbidden_is_rejected_with_blocked_status` integration test
# in tools/quality/tests/http_smoke.rs. The refusal IS audited (appended
# to the ledger), so a future audit can confirm the policy was applied.
# E7 is now covered by
# ui/e2e/synthetic-workspace-cleanup-budget.spec.ts (≤4 API calls,
# ≤3 s wall-clock, idempotent, origin-targeted).

# Cross-cutting concurrency stories — high product value, harness gap.
# X-G2-S1 is now covered by
# ui/e2e/stories/cross-cutting/x-g2-s1-three-actors-converge-after-extract.spec.ts.
# The spec proves the durable cross-actor convergence contract (alice + bob
# present, carol cold-joins). Specific extract field values are not asserted
# because the default-bucket test environment has no LLM provider keys —
# the catalog claim is about CONVERGENCE, not extract content.
# X-G6-S1 is now covered by
# ui/e2e/stories/cross-cutting/x-g6-s1-tailor-while-friend-edits.spec.ts.
# Local proof of the broker-level invariant: concurrent tailored.snapshot
# (job sub-doc) and cv_profile.replace (quarry sub-doc) both commit cleanly
# without corrupting each other. The 20 s LLM-delay scope lives in
# journey-real/ behind an explicit opt-in (live providers).
# X-G6-S2 is now covered by
# ui/e2e/stories/cross-cutting/x-g6-s2-agent-tailors-while-user-offline.spec.ts.
# Broker-level proof: user prepares an op offline (held in test-space, not
# POSTed), agent writes tailored.snapshot, user replays. Both writes co-exist
# and are durably present. The full SDK IDB-queue driving from a browser
# tab is a presentation-layer concern; the underlying broker contract
# proven here is what makes that path safe.
# X-G10-S1 is now covered by ui/e2e/persistence/batch-atomicity.spec.ts.
# The post-commit single-snapshot read is the strongest practical proof of
# "peers see consistent state" because the D1 batch is one transaction; any
# observer reading at any moment either sees the whole batch or doesn't see
# any of it.

# Multi-device leader-election stories.
# P6-G5-S1 is now covered by
# ui/e2e/stories/device/p6-g5-s1-multi-device-kanban-drag.spec.ts.
# Same convergence shape as P2-G5-S1; the "two devices" aspect is
# presentation-layer (a phone is a smaller viewport, not a different data
# source), so the durable-projection convergence within 2 s is the right
# proof. Replace this with a real two-browser-context spec if/when phone
# viewport rendering becomes a separate concern.
# P6-G6-S2 is now covered by
# ui/e2e/stories/device/p6-g6-s2-leader-election-prevents-double-claim.spec.ts
# (live two-tab proof that only one claim wins). Do not re-waive without
# explaining why that spec is no longer adequate.
Source: wiki/content/stories/story-coverage-waivers.md