Auth operations runbook

Day-to-day operations for the CV + jobcache auth stack. Pairs with PRIVACY-POSTURE (the design) and auth-pick (the why).

Token catalogue

Two completely separate mechanisms, both surface as Authorization: Bearer …:

Mode A — Static service tokens (@cv/auth-bearer)

Stateless. SHA-256-hashed registry in a wrangler / Render env var. No DB.

Mode B — User-mintable API keys (Better Auth apiKey plugin)

DB-backed. Per-user, scoped, revocable. Sent as x-api-key: <key>.

Common operations

Mint a static service token

# 1. Generate a random opaque token
TOKEN=$(node -e 'console.log("cv_" + require("crypto").randomBytes(20).toString("base64url"))')
echo "Token (give to caller, never to git):"
echo "$TOKEN"

# 2. Hash it (this is what goes into the env var)
HASH=$(node -e 'console.log(require("crypto").createHash("sha256").update(process.argv[1]).digest("hex"))' "$TOKEN")

# 3. Build the registry JSON (or append to existing)
cat <<EOF
{
  "$HASH": {
    "label": "ci-ingest@circleci",
    "scopes": ["ci-ingest"]
  }
}
EOF

# 4. Push to the worker
wrangler secret put QA_TOKENS_JSON --env production
# (paste the entire JSON object — Cloudflare overwrites, so include any
# existing entries you want to keep)

Give $TOKEN to the caller (CI runner, agent, CLI). They send Authorization: Bearer $TOKEN.

Mint a user API key (jobcache)

User goes to their settings page in jobcache and clicks "Create API key" — under the hood:

# Equivalent direct call (for testing):
curl -X POST https://api.jobcache.corbet.ch/api/auth/api-key/create \
  -H "Cookie: <user-session-cookie>" \
  -H "Content-Type: application/json" \
  -d '{ "name": "my-claude-agent" }'
# Response: { "key": "<plaintext-once>", "id": "...", "name": "my-claude-agent" }

The plaintext key is shown ONCE. User stores it in their agent's config. Future requests send x-api-key: <plaintext>.

Revoke a key

Static service token:

# Remove the hash entry from the registry JSON, then redeploy the secret.
wrangler secret put QA_TOKENS_JSON --env production
# (paste the JSON with the revoked entry removed)

The revocation is instant because the per-process cache keys on the env-var string — when wrangler pushes a new secret, the worker re-reads it on next cold start. To force-revoke immediately: also bump a worker version (wrangler versions deploy).

User API key (jobcache):

# User clicks "Revoke" in settings, or:
curl -X POST https://api.jobcache.corbet.ch/api/auth/api-key/delete \
  -H "Cookie: <user-session-cookie>" \
  -H "Content-Type: application/json" \
  -d '{ "keyId": "<api-key-id>" }'

Revoke updates apikey.revoked_at and all subsequent verifications fail.

Rotate all tokens (suspected leak)

1. Static service: Generate new token, push registry with NEW hash, give caller the new token. Then push registry without the old hash. (Two-step lets the caller cut over without downtime.)
2. User API keys: Better Auth doesn't have a bulk rotate. Either (a) admin queries the DB and revokes all apikey rows for a user (UPDATE apikey SET revoked_at = NOW() WHERE user_id = ?), or (b) tell the user to rotate themselves.

Add a new admin

For CV admin (passkey-only on auth.careervector.corbet.ch):

1. Provision the admin's user row with role='admin':

   INSERT INTO user (id, role, created_at)
   VALUES (gen_random_uuid(), 'admin', NOW())
   RETURNING id;
   

2. Generate an enrollment link with a one-time token (deferred — careervector-auth worker doesn't have this UI yet, see spike state).
3. Admin visits the link in their browser, enrolls one or more passkeys.
4. They sign in via the same passkey on qa.careervector.corbet.ch / ops.careervector.corbet.ch (cookie crosses .careervector.corbet.ch).

Debug a 401

1. Check the request actually sent the header.

   curl -v https://api.qa.careervector.corbet.ch/v1/runs/foo 2>&1 | grep -i "^> "
   

   Confirm Authorization: Bearer … appears in the outgoing headers. The #1 cause of 401s in this stack is missing headers, not bad tokens.
2. Check the token is in the registry.

   echo -n "$TOKEN" | sha256sum
   # Compare with the registry JSON
   

3. Check the token has the required scope. A ci-ingest-scoped token can't call mcp-trigger endpoints.
4. Check the env var is on the right env. wrangler secret list --env production should show QA_TOKENS_JSON. Easy footgun: pushed to default env instead of production.
5. Check the worker has been deployed since the secret was set. Wrangler secrets take effect on the next deploy / version push. wrangler versions list --env production.

Wipe a compromised account

User reports their account is taken over:

# 1. Admin runs (in jobcache Cockroach):
UPDATE apikey SET revoked_at = NOW() WHERE user_id = '<user-id>';
DELETE FROM session WHERE userId = '<user-id>';
DELETE FROM account WHERE userId = '<user-id>';
DELETE FROM passkey WHERE userId = '<user-id>';

# 2. Tell the user to enrol new passkeys via the sign-up flow. Their
#    user_id stays the same; only the credentials are wiped.

We don't have a "force-logout" button in the UI yet — the cascade above is the manual version.

Where things live

Quick health checks

# Is the careervector-mcp gate working?
curl -s -o /dev/null -w "%{http_code}\n" https://mcp.careervector.corbet.ch/
# Expect: 200 (GET is public)

curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize"}' \
  https://mcp.careervector.corbet.ch/mcp
# Expect: 401 without a Bearer (good — gate is closed)

# Is the qa-api gate working?
curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST https://api.qa.careervector.corbet.ch/v1/tests/affected-by \
  -H "Content-Type: application/json" \
  -d '{"paths":["foo.ts"]}'
# Expect: 401 without a Bearer

Known footguns

• The cf-access-jwt-assertion path is GONE in ops/api. If you migrate the worker behind Cloudflare Access in the future, the JWT signature must be properly validated (use @cloudflare/access-jwt-validator or equivalent). Header presence is NOT proof of auth.
• Memory adapter table-init. When testing Better Auth, pre-create the tables in the in-memory DB: db.user = [], db.session = [], db.account = [], db.verification = [], db.passkey = [], db.apikey = []. findOne throws for missing tables; only create lazily inits.
• Passkey register-options is GET, not POST. Easy to get wrong.
• api-key plugin has no /verify endpoint. Send x-api-key on protected routes; the plugin's middleware validates.
• The api-key plugin's model name is apikey (lowercase). Not apiKey. Better Auth's naming is inconsistent across plugins.
• Token rotation: push the NEW hash before removing the OLD one. Otherwise CI calling with the old token gets 401 during the gap.
• Auto-commit timer eats uncommitted work. If dotkeeper or similar runs in this repo, commit aggressively. The reflog will show reset: moving to HEAD + checkout: moving from <branch> to main — that's the smoking gun. (Learned the hard way during this spike — hours 4-5 had to be re-applied from the chat history once.)