JobCache Implementation Checkpoint

This checkpoint captures the current implementation slice so parallel agents can move quickly without reopening settled model decisions. Treat it as the coordination page for active JobCache work until a newer checkpoint replaces it.

Current Shape

CareerVector and JobCache share one sparse payload tree:

role -> ad -> workspace job

• role is the URL-family grouping above one or more language-specific ads.
• ad is the public URL x language object in JobCache.
• workspace job is workspace-local and subjective.
• Sparse payload fields use the same vocabulary at every level. Metadata needed to run the system stays outside the payload vocabulary.
• Submitted Observations anchor to one ad_id; they do not address roles or workspace jobs.

There are two ingest directions:

• CareerVector bottom-up: a workspace URL is added, CareerVector checks JobCache by URL x language, then queues ensure_ad and safe submit_observation commands when public fields are available.
• JobCache top-down: planner creates tasks, devices lease tasks, devices submit Observations, and submitObservation applies the shared writer.

The common interface is the database and shared package contracts, not a JobCache user interface.

Canonical Words

• Observation: full submitted envelope for public ad data.
• Adapter: language-neutral dock contract for one platform/source shape.
• Device: durable enrolled runtime/place where work can run and advertise capabilities.
• Scraper: Adapter plus Device while executing a scrape task.
• Env: runtime capability bundle passed through the Adapter dock.
• submitObservation: server-side write gate for submitted Observations.

Avoid: artifact for JobCache payloads, job-corpus, SourceAdapter, AdapterContext, fetchOne, scrapeObservation, submitTaskResult, and parallel payload names.

Implemented Now

• lib/jobcache is the shared TS package for ids, contracts, Adapter/Env, and Observation helpers.
• adIdFromUrlLanguage(url, language) is canonical. Every ad identity is URL x language; und is an explicit unknown-language bucket in that same identity scheme.
• roleIdFromUrl(url) is canonical for the role above those ads. The German, French, English, and unknown-language ads for the same canonical URL share that role id.
• submitObservation derives the canonical ad id from normalized URL x language server-side before writing, so a bad device-supplied ad_id cannot collide with the URL-language uniqueness index.
• submitObservation writes current ad payload cells, propagates safe changed ad fields to the URL-family role, updates lightweight observation state, updates device trust counters, and creates follow-up verification tasks on fresh conflicts.
• The hot path derives the URL-family role id with roleIdFromUrl(url), keeps the ad linked to that role, and performs only cheap ad_propagation for safe changed fields. Semantic role reduction, cross-language aggregation, and company/title clustering are deferred async database work.
• Observation-carried object_refs, chunk_refs, annotations, search_chunks, and embeddings are persisted into hot-index tables.
• Hot-index tables are part of ensureFactTreeSchema(), not only migration files.
• Duplicate broker task enqueue uses insert-or-readback: repeated enqueue returns the existing task_id, input_hash, and state without a no-op row rewrite.
• Lease selection first expires a bounded oldest-first batch of overdue active leases, then picks retryable work.
• Identity and hot-index upserts use no-op guards where possible so repeated role, ad, object, chunk, annotation, and embedding rows do not churn writes.
• Repeated evidence writes use insert-or-read instead of rewriting the same payload for an existing content hash.
• Role read paths use the URL-family role link plus named async reducer rows only, so stale direct/adapter role rows cannot leak as inherited payload.
• CareerVector shared lookup forwards the language hint through /api/ads/lookup; JobCache lookup tries exact URL x language first and only falls back to und.

Guardrails

• Server code may validate schema, leases, policy, references, hashes, trust, and DB writes.
• Server code must not fetch pages, run Adapters, chunk, enrich, embed, call an LLM, geocode, or read object-store blobs for device task output.
• Devices never receive Cockroach credentials or permanent object-store credentials.
• Agreement should avoid rewriting large payload rows. Disagreement may rewrite current cells and tiny state.
• Duplicate and retry paths should return existing rows or update tiny lease state, not refresh identity rows just for freshness.

Active Work Lanes

Use disjoint ownership. Parallel work is safe only when each lane stays inside its files and treats the critical section below as integrator-owned.

Critical Section

Do not edit these in parallel. A single integrator should own each change and re-run all affected lane validations:

• lib/jobcache/src/contract.ts
• lib/jobcache/src/ids.ts
• lib/jobcache/src/adapter.ts
• jobcache/shared/src/fact-tree.ts
• jobcache/shared/src/schema.ts
• jobcache/shared/src/device-contract.ts
• jobcache/interface/src/api/commands/submit-observation.ts
• jobcache/interface/src/api/commands/ensure-ad.ts
• jobcache/interface/src/api/device-control.ts
• jobcache/interface/src/api/planner/**
• jobcache/migrations/**
• lib/domain/src/jobcache.ts
• lib/server/src/workspace-subdoc.ts
• lib/server/src/workspace-ops.ts
• jobcache/ingest/src/adapters/registry.ts
• crates/** and Cargo files

If a lane needs one of these files, stop and claim the critical section in the handoff before editing.

Implemented Proofs

• The broker/device flywheel has an end-to-end fake-device proof from source/binding planning through lease, submitted Observation, task closeout, hot-index writes, device trust, and follow-up verification task.
• The CareerVector bottom-up path proves that jobcache_commands.submit_observation can be committed with the private job.create or job.update, then drained through the same shared submitObservation writer.
• Adapter tests prove concrete source Adapters emit valid Observation envelopes. Historical scrapers/ implementation-path names remain only as code paths, not model vocabulary.
• Rust parity code mirrors URL identity and the Adapter/Env/Observation boundary against TS golden vectors before any native execution path becomes load bearing.
• Ops, status, and QA expose tasks, leases, submissions, stale leases, conflicts, trust, missing object refs, and verify tasks.
• QA proves duplicate enqueue readback, bounded lease cleanup, and no-op-gated identity rows as RU regression checks whenever the broker changes.
• Role reduction is not the next slice. URL-family role membership already comes from roleIdFromUrl(url) and cheap ad_propagation; cross-language payload aggregation and company/title clustering should run later as async DB work. Do not block the broker/device proof on them.

Validation Baseline

For this docs checkpoint and its wiki links:

bun --filter @cv/wiki check
bun --filter @cv/jobcache-wiki check

Also grep the touched docs for the terms listed in Canonical Words -> Avoid. Hits should appear only in explicit Avoid lists or historical-path notes.

Recent implementation green checks:

bun --filter @cv/jobcache-shared test && bun --filter @cv/jobcache-shared check
bun --filter @cv/jobcache-interface test && bun --filter @cv/jobcache-interface check
bun --filter @cv/jobcache test && bun --filter @cv/jobcache check
bun --filter @cv/jobcache-ingest test && bun --filter @cv/jobcache-ingest check
bun --filter @cv/domain test && bun --filter @cv/domain check
bun --filter @cv/server test -- src/jobcache-lookup.test.ts src/workspace-ops.test.ts
bun --filter @cv/server check
bun --filter @cv/mutations test -- src/apply.test.ts && bun --filter @cv/mutations check
bun --filter @cv/api check
git diff --check

Minimum lane-specific checks:

# Broker loop
bun --filter @cv/jobcache-interface test -- src/api/device-control.test.ts src/api/planner/source-bindings.test.ts src/api/commands/submit-observation.test.ts
bun --filter @cv/jobcache-interface check

# Device harness
bun --filter @cv/jobcache-shared test -- src/device-client.test.ts src/device-contract.test.ts
bun --filter @cv/jobcache-shared check

# CareerVector bridge
bun --filter @cv/domain test && bun --filter @cv/domain check
bun --filter @cv/server test -- src/jobcache-lookup.test.ts src/workspace-ops.test.ts
bun --filter @cv/server check

# Adapter contract
bun --filter @cv/jobcache-ingest test
bun --filter @cv/jobcache-ingest check

# Root and JobCache wiki
bun --filter @cv/wiki check
bun --filter @cv/jobcache-wiki check

Warnings seen during some runs:

• Node DEP0205 in @cv/jobcache-interface.
• Bun localStorage experimental warning in some domain/server tests.

Both warnings were non-fatal.

Next Best Slice

The highest-value next proof is the broker/device flywheel:

source/binding -> task -> lease -> device Observation -> submitObservation
  -> current cells/hot index/trust/follow-up task

Once that loop is pinned with fake devices and tests, Adapter work can proceed as plug-in implementation rather than architecture work.