auth-spike learning log

A running journal of gotchas, surprises, "wait, that worked?" moments, and dead ends encountered while building out the auth architecture on this branch. The point of this log is the journey, not the destination — if something turned out to be harder, weirder, or simpler than expected, write it down.

Format: chronological. Each entry is ## YYYY-MM-DD HH:MM — title with free-form notes. Cross-link to commits when relevant.

2026-05-18 — branch created, starting from main 41074bba

Two wiki docs already live on main: architecture/PRIVACY-POSTURE.md (the data-minimization principle) and studies/auth-pick/README.md (the research trail). The branch is for the implementation only.

The plan has eight tasks (#47–#54). Steps 1–3 are mechanical "extract and unify the bearer dialects we already have." Step 4 plugs a small security hole. Steps 5–7 are the genuinely new work: Better Auth in jobcache-interface, a new careervector-auth worker for CV admin, and user-mintable api_keys.

2026-05-18 — step 1: extract lib/auth-bearer/ (commit ba89b780)

Surprise 1: two different "Caller" shapes already in qa-platform. api-core uses {token_name, scopes: [...]} (array), mcp-core uses {label?, scope: "..."} (single string). I'd assumed they were the same. The extract had to accept BOTH input shapes and normalise to {label, scopes: [...]}. Side benefit: no production env-var rotation is needed when downstream consumers migrate — the parser handles both.

Gotcha: my first requireAuth impl was wrong. I created the makeRequireAuth Hono middleware INSIDE the request handler instead of at module load time, which would have re-parsed the env var on every request and discarded the per-process cache. Caught it before commit by re-reading the code; fixed by extracting the underlying primitives (extractBearer, sha256Hex, parseRegistry) directly in the qa-platform adapter file and keeping the cache as module-scope state there. Lesson: Hono middleware factories that close over state must be called once, not per-request.

Strict-mode type issue surfaced two commits later (in step 3). noUncheckedIndexedAccess consumers (jobcache-ingest, which has stricter tsconfig) rejected view[i].toString(...) and match[1] because TS typed both as possibly-undefined despite the guard contexts. Fixed with a narrow cast for the loop-indexed case and a ?? null for the regex capture. Worth standardising the strict-mode preferences across the monorepo eventually.

2026-05-18 — step 2: ops migrations (commit 7feeb46e)

Found a real security hole during migration. ops/api/src/lib/auth.ts had three acceptance paths; the first one was "presence of cf-access-jwt-assertion header → trusted." The JWT signature was NEVER validated. If anyone ever bypassed the CF edge (or if the worker were moved to a non-CF host), the path was spoofable by setting the header to anything. Removed entirely in step 2 — the bearer + cookie paths cover the same audience without trusting an unverified token.

Tests failed first run because of bearer-vs-registry mismatch. Tests expect ADMIN_TOKEN bearer to work, but I'd moved bearer to the hashed registry. Fixed by adding a transitional ADMIN_TOKEN fallback path inside the bearer branch (matches what I did for ops/mcp). Lesson: when adding a new env var that replaces an old one, keep BOTH working during transition. The migration cost shows up as a fallback branch in code; the production rotation cost is zero.

The isAdminRequest function was dead code. Grepped across the whole monorepo + svelte files; only the declaration referenced it. Deleted it.

2026-05-18 — step 3: jobcache verifier unification (commit 12e99e0c)

@cv/auth-bearer works in both CF Workers AND Node. jobcache-ingest runs on Render Node, not Workers. The library uses crypto.subtle and TextEncoder — both globally available in Node 19+, no shims needed. The @cloudflare/workers-types package in the lib's devDependencies is just type augmentation; Node consumers ignore it at runtime. Universal lib without effort.

Order of 503 unconfigured vs 401 unauthorized matters. First test failure: I checked "is there a bearer header?" before "is anything configured?" — which meant misconfigured workers returned 401 (looks like a credential problem) instead of 503 (looks like an ops problem). The test caught this. Reordered the checks. Lesson: HTTP status codes are operator signals; pick the one that points at the right problem.

authorizeJobcacheControl is now async. It was sync (plain string compare). WebCrypto's digest() is async, so the new signature is too. Only one call site needed await added — but it could have been many. Lucky that the JOBCACHE_CONTROL_TOKEN was used in few places.

2026-05-18 — step 4: careervector-mcp gating (commit 2380aeeb)

The CV MCP had ZERO auth before this commit. Bigger than I assumed. The landing page said "Workspace ID is the access credential" — but POST /mcp accepted tools/call from anyone with no Bearer at all. Any caller who knew a workspace nanoid could mutate state. Plugged with a hashed-registry lookup of CV_MCP_TOKENS_JSON, fail-closed when the env var is empty.

Sync test helpers vs async hash function — first real friction point. makeFakeEnv() in mcp/test/fakes.ts is sync (function makeFakeEnv(): { env: Env, ... }). Making it async to compute the test token's hash on the fly would have forced every consuming test to add await. Instead I pre-computed the SHA-256 in a one-shot Node command and hardcoded both the bearer and its hash as constants. Sync stays sync. Documented the re-computation command in a comment so the next person isn't lost.

Lesson: the tail of an async primitive can hit a lot of call sites. Pre-computing or stubbing the result is often cheaper than async-ing the world.

4 new auth-targeted tests added. Missing-bearer, wrong-bearer, no-tokens-configured, and "GET / stays public even with no tokens." The last one was important — wanted to confirm the discovery surface stays agent-reachable without a token, so an unknown agent can probe what's there before asking for one.

2026-05-18 — step 5: Better Auth spike in jobcache-interface (in progress)

Setup was anticlimactically smooth. bun add better-auth brought 50 deps (1 ms heavier dep tree than I expected — kysely, jose, @noble/hashes, @noble/ciphers, defu, nanostores, better-fetch). Built-in adapters ship for Drizzle, Kysely, Prisma, Mongo, memory. Their peerDependencies declare pg (node-postgres), not postgres (postgres-js — what jobcache already has). I added pg + @types/pg as a second DB client just for Better Auth; the two clients are independent and small.

Big-picture pick: Kysely-via-pg, not Drizzle. Less ceremony, no generated TypeScript schema, no separate migration files. Trade: Drizzle is the more popular long-term choice in TS land. We can swap.

OAuth profile mapping — discarding PII at the import boundary. Per PRIVACY-POSTURE, we DON'T want to store email/name. Better Auth's mapProfileToUser() hook fires before any DB write; setting { email: "", name: "" } makes the rows boring. Open question I haven't verified: does Better Auth's internal schema require email to be unique? If yes, two OAuth-only users would collide on an empty string. Likely need a custom user schema to drop the constraint. Logged as TODO.

Trusted origins + cross-subdomain cookies. Better Auth has built-in crossSubDomainCookies config — set enabled: true and domain: ".jobcache.corbet.ch" and it issues cookies that sibling subdomains can read. Reads cleanly without me having to learn cookie-internals.

The module imports without crashing even with no real DB. New pg Pool is lazy — it doesn't connect on construction, only on first query. So auth.handler is callable in tests without a running Postgres. The handler routes a benign GET to a Response (we got 404 from Better Auth's internal router). 39 tests in jobcache-interface stay green.

What I haven't done yet (deliberately deferred for the spike):

• Mount auth.handler into the apiApp Hono surface so /api/auth/* actually serves
• Generate the DB schema (Better Auth's CLI: npx @better-auth/cli generate)
• Test against actual Cockroach to find PG↔CRDB feature gaps
• Wire OAuth provider creds (Google / Microsoft cloud-side setup required first)
• SK client-side integration (sign-in buttons, session readers)
• Session middleware on protected SK routes
• Verify the empty-email user uniqueness behaviour

The library is friendly enough that the import + handler-callable proof is real signal that this path works. The unknowns are operational (provisioning, DB schema cross-compatibility, OAuth client IDs) rather than architectural.

2026-05-18 — step 6: careervector-auth worker scaffold (commit 4f672bed)

Better Auth + Kysely + kysely-d1 composes — at type-check time. The chain is: Better Auth's database parameter accepts a Kysely instance. Kysely accepts any dialect implementing its Dialect interface. kysely-d1 (MIT, v0.4.0, 14 KB, no deps) is a community dialect that wraps a D1Database binding. Plugging them together typechecked on the first try; one as any cast through Better Auth's loose database typing was needed.

TS friction with Better Auth's loose typing. Better Auth's BetterAuthOptions.database is typed as an enormous union (Kysely, Drizzle, Prisma, Mongo, …) and the user's specific Kysely instance often doesn't match the union strictly. The escape hatch is as any on the two call sites. Annoying but not blocking — Better Auth's generated schema typings (post @better-auth/cli generate) would solve this properly.

D1 binding has no real database_id in the wrangler.toml. I left database_id = "00000000-..." as a placeholder; before any real deploy this would be set with wrangler d1 create careervector-auth. The spike never deploys. The boot test smoke-checks the worker's discovery + healthz routes only — Better Auth's actual sign-in flow needs a real D1.

2026-05-18 — step 7: api_keys sketch (commit d5409cb9)

Two-store bearer is a clean architecture in TS. The middleware flow: try env-var registry → DB-backed api_keys → fail. The Caller shape is the same in both cases ({label, scopes[]}) with the DB path additionally carrying user_id. One bearer header, two backing stores, indistinguishable to the caller.

Token format choices. jck_<env>_<entropy> with <env> ∈ {p, d} keeps dev keys from accidentally working in prod (prefix lookup just misses). Base32 over base64 for case-insensitivity in copy/paste flows. 32 bytes of entropy = 256 bits. Storing the first 8 chars unhashed in key_prefix lets the UI show jck_p_ABCD**** for identification without exposing the secret. The remaining hash is what gets looked up.

Fire-and-forget last_used_at touch. void sql\UPDATE …`.catch(...)` keeps the response path fast. Worst case: one slightly-stale timestamp or one missed update on an extremely-hot key. Both are fine for an audit trail.

The DDL targets Cockroach directly. gen_random_uuid(), TEXT[] NOT NULL, TIMESTAMPTZ all work in Cockroach today. Partial index on WHERE revoked_at IS NULL may need verification — Cockroach supports partial indexes but some shape variations exist.

Overall takeaways

After 8 commits, ~7 hours of focused work on the auth-spike branch:

What turned out easier than expected:

• Extracting lib/auth-bearer/ from lib/qa-platform. The two slightly- different existing dialects ({token_name, scopes[]} vs {label, scope}) were absorbable into one parser with a single normalization pass. No prod env-var rotation needed.
• Better Auth as a library import. The package installs cleanly, the config is a single function call, the handler is a (Request) => Promise<Response> shape that drops straight into any framework.
• kysely-d1 for Workers + D1. Community dialect, MIT, 14 KB, no deps. Just works.
• Universal lib code. @cv/auth-bearer runs in both CF Workers (qa-mcp, ops-mcp, careervector-mcp, careervector-auth) and Node (jobcache-ingest, jobcache-interface) with no shims — crypto.subtle and TextEncoder are global everywhere we care.

What turned out harder than expected:

• The ops/api auth file was harbouring a real security bug: a cf-access-jwt-assertion presence check with no signature validation. Discovered it during the migration audit, not by looking for bugs.
• Sync ↔ async hash split caused test-suite friction in the qa-mcp + cv-mcp paths. The "pre-compute the test bearer's SHA-256 once and hardcode" pattern was the right escape hatch but not obvious upfront.
• Better Auth's database parameter is loosely typed — needed as any in two places. Their schema-gen CLI would resolve this properly, but the spike didn't get that far.
• The careervector-mcp had ZERO auth before this branch. Bigger gap than I'd assumed from the architecture docs.

What surprised me:

• The "no vendor lock-in, move between Cloud and self-hosted anytime" marketing line on Hanko Cloud was substantially less polished than advertised. Their own docs say "with our help" — i.e., sales call. Same story on Logto Cloud (different surface, same shape). Self-hosting from day one is the only genuinely-zero-migration path.
• Logto OSS's "single admin only" limit is a real constraint not visible from marketing. WorkOS's $99/mo custom-domain charge is similarly weird rent-seeking. Reading the actual pricing pages mattered.
• The transitional fallback pattern (old plain-string compare PLUS new hashed registry) is doing more of the heavy lifting than I expected. It means no prod env-var rotation, no caller migration, no downtime — consumers cut over individually on their own schedule.

What's still unknown:

• Cockroach vs Postgres compatibility for Better Auth's schema. The CLI generates DDL targeting stock PG; some features (uuid functions, array syntax, partial indexes) may need tweaks. Spike didn't get this far.
• WebAuthn relying-party-ID handling when serving auth on auth.careervector.corbet.ch but expecting passkey signatures usable from qa.careervector.corbet.ch. The two are sibling subdomains — spec-wise this should work with rpId: ".careervector.corbet.ch" but worth verifying with a real enrollment ceremony.
• Better Auth's empty-email-user uniqueness behaviour. If their default schema declares email UNIQUE NOT NULL, an OAuth-only user with email: "" would collide with all other OAuth-only users. Likely need a custom user schema or a uniqueness override.

What I'd do differently next time:

• Audit existing auth surfaces FIRST. The unverified-JWT bug + the ungated MCP worker were two real findings discovered mid-migration. Should have been a single targeted audit at the start.
• Sketch the test-bearer helper pattern before changing the first production worker. The pre-compute-sha256-and-hardcode trick took two rounds to land cleanly.
• Don't write the wiki study BEFORE doing the spike. The auth-pick study shipped with the wrong human-auth picks (Google + Microsoft OAuth) before the conversation refined it to "passkey-first + no email." Doc-led design here would have shipped wrong; spike-led design got it right.

2026-05-18 — Hours 1-6 deep-dive (post-design lock)

After the design landed, Julian gave a 6-hour "exploratory coding" mandate — go deep, find gotchas, become a reliable expert. The deliverable was LEARNING, not a working production system. Below: everything that turned out to be true / false / surprising in the second pass.

Hour 1 — Better Auth flows against memory adapter

6 tests. GOTCHA 1: memory adapter doesn't lazy-init tables. findOne throws "Model X not found" if db[model] is missing; only create lazily inits. Sign-up calls findOne(email) FIRST to dedupe, so the table must already be []. Pre-init: user, session, account, verification (core). GOTCHA 2: /api/auth/get-session returns JSON null after sign-out, not {user: null}. Easy assertion mistake.

Hour 2 — Passkey + api-key plugin ceremonies

8 tests, 4 gotchas:

• Passkey plugin's /passkey/generate-register-options is GET, not POST.
• Passkey register requires freshSessionMiddleware — needs a session minted recently. Fresh sign-up provides one.
• api-key plugin has NO public /verify route. Only create/get/list/update/delete. Verification happens via the middleware that reads x-api-key header on routes.
• api-key plugin's memory-adapter table name is apikey (lowercase). Not apiKey. Better Auth's naming is inconsistent across plugins.

Hour 3 — Hono integration

5 tests. Mounted app.all("/api/auth/*", c => auth.handler(c.req.raw)). New requireSession() and attachSession() middleware in session-middleware.ts. Clean — no gotchas at this layer; the friction lives one level down in plugins.

Hour 4 — SvelteKit hooks.server.ts + auth-guard.ts

Hydrates event.locals.session on every request via Better Auth's auth.api.getSession({ headers }). Route-gates /account/* etc with a redirect that preserves the returnTo destination. Already-signed-in users on /sign-in bounce home.

Hour 5 — Cross-subdomain cookies

5 tests. Confirmed: Domain=.jobcache.corbet.ch is set on the cookie, sibling subdomain reads work, isolation between jobcache and CV is correct (two separate auth instances reject each other's cookies). The cross-subdomain property is load-bearing for the architecture.

Hour 6 — Compliance endpoints + ops runbook

6 tests on /v1/me/export + /v1/me/delete. The export goes through Better Auth's auth.$context.adapter to read user + sessions + accounts

• plugin tables. Delete cascades manually through plugin tables then user (needed because memory adapter has no FK; real Cockroach uses ON DELETE CASCADE). GOTCHA: Better Auth's built-in auth.api.deleteUser requires email confirmation; bypassed for privacy-minimized (no-email) mode.

Bonus hour — PII strip + rate-limit + bearer-vs-session

After hour 6, kept pushing for "reliable expert" status:

Hour 7 — PII strip (stripPiiHeaders)

• CRITICAL: Better Auth stores BOTH ipAddress AND userAgent on every session row by default.
• advanced.ipAddress.disableIpTracking = true disables IP capture.
• No equivalent flag exists for User-Agent in 1.6.11.
• Solution: a request shim that strips user-agent before handing the request to auth.handler. New strip-pii.ts does this.
• 6 tests including a NEGATIVE CONTROL that proves UA leaks in WITHOUT the shim. The negative control is defensive: it tells future maintainers WHY the shim matters.

Hour 8 — Session lifecycle + multi-device

• Default Max-Age is 7 days (604800s). Session row's expiresAt is a Date set the same distance out.
• Expired session reads as null, not 401. Application decides.
• Multi-device sessions coexist. Sign-out is per-cookie, not per-user. "Log out everywhere" must be built explicitly.

Hour 9 — Bearer + session precedence

• 2 tests answered the real production question.
• CONFIRMED: When BOTH cookie + x-api-key arrive, the cookie WINS.
• API key alone on /api/auth/get-session returns 200 + null — no synthesized session.
• Implication: api-keys are for ROUTES, not for the session-shape endpoint.

Hour 10 — Rate limiting — CORRECTION

• I had said earlier "Better Auth doesn't ship rate limiting." That was wrong.
• rateLimit.enabled defaults to TRUE in production, FALSE in dev.
• rateLimit.storage defaults to "memory" — survives a Node service's lifetime, but NOT CF Workers' invocation boundaries. For Workers configure "database" so the rate-limit rows live in the worker's DB.
• 4 tests verify per-IP buckets, window/max, and the disabled-equals-unlimited behavior.

Final state

14 packages, 625 tests pass. New tests added this spike sub-session:

Files added to the production stack:

• lib/auth-bearer/ (extracted, prod-ready) — 37 tests
• careervector-auth/ (new worker scaffold) — 2 tests, spike-grade
• jobcache/interface/src/auth/index.ts — production config with PII guards
• jobcache/interface/src/auth/session-middleware.ts — Hono middleware factories
• jobcache/interface/src/auth/strip-pii.ts — Request PII shim
• jobcache/interface/src/auth/me-endpoints.ts — GDPR export + delete
• jobcache/interface/src/auth/api-keys.ts — DB-backed user-mintable keys (sketch)
• jobcache/interface/src/ui/hooks.server.ts — SK session hydration
• jobcache/interface/src/ui/lib/auth-guard.ts — requireSession() for SK servers
• wiki/content/runbooks/AUTH-OPS.md — mint/revoke/rotate/debug/wipe runbook
• wiki/content/architecture/PRIVACY-POSTURE.md — data-minimization principle (landed pre-spike)
• wiki/content/studies/auth-pick/README.md — research trail (synced to final design)

What I'd add if given another 6 hours

• Real Cockroach pglite test that runs Better Auth's migrations and confirms schema works
• Miniflare test of the careervector-auth worker with a real D1
• Svelte sign-in.svelte using Better Auth's client SDK
• WorkOS Connections-as-a-Service spike to confirm the SAML escape valve is one-day work
• Actually deploy the spike to staging and click through a sign-in
• A SOC2/Schrems II-ready privacy policy + DPA template

Mid-spike incident — lost work

At one point during this sub-session, an auto-commit timer (dotkeeper or similar) ran git reset --hard HEAD + git checkout main while I had uncommitted hours-4-5 work. Files vanished from disk; commits already pushed to origin/auth-spike were safe but the in-flight changes were gone. Recovered by re-applying from chat history.

Lesson logged in AUTH-OPS.md: when an auto-commit process runs in the same repo, commit aggressively — every small chunk, not at "logical end-of-task" boundaries. The reflog will show the smoking gun (reset: moving to HEAD followed by a checkout-to-main).