Architecture

JobCache is a bun-workspace tree with an observer-private App zone and a shared Core zone. api/ and mcp/ are live Cloudflare Workers; interface/ is the protected Core HTTP process; shared/ is the Core data layer; and ingest/ contains trusted device/local/operator harnesses rather than a central scraper. The canonical reference is jobcache/README.md in the repo; this page mirrors the current stack and reasoning.

Current active-lane coordination lives in the root wiki SCRAPER-FLEET and CONTROL-PLANE. Use them for critical-section ownership, remaining gaps, and the fleet / control-plane contracts before starting parallel implementation work. (The older working-memory implementation checkpoint froze 2026-05-29 — pre-CrateDB, pre-orchestrator — and is historical only.)

Stack

  • Language: TypeScript on Node.js (esm).
  • Core service: interface/src/index.ts, bundled for Node 24 and live at the protected Alwaysdata /core origin. It exposes corpus reads and independently authenticated control/lifecycle operations. It owns no App session route.
  • App edge: api/ owns opaque session capabilities and private D1 state; mcp/ is its thin secretless peer. Both are live at the branded public hosts.
  • UI edge: ui/ is the Cloudflare SvelteKit product shell. It shares @cv/public-ui, @cv/jobcache-participant, and @cv/link-realtime with CareerVector rather than copying shell status, contribution, derivative-link, or presence mechanics.
  • Ingest harness: ingest/src/index.ts. Trusted local/device/operator harness for adapter execution. It is not provisioned as a central Render scraper.
  • Shared package: shared/src/. Postgres client, schema, fact tree, snapshots, metrics, OTel/Sentry bootstrap. Both services import from @cv/jobcache-shared.
  • Hot DB: CrateDB Cloud CRFREE (West Europe, Azure; pg-wire only — no JSONB/txns/FK, dialect contract in jobcache/shared/src/pg-options.ts). Cut over from CockroachDB Serverless 2026-06-08; the Cockroach cluster is dormant, rollback-only.
  • Cold evidence: online S3-compatible object storage; B2 is the current default for replaceable raw HTML/API/evidence blobs.
  • Hosting: Cloudflare Workers + EU D1 for App/MCP; one protected Alwaysdata user program for Core. The old Render service remains suspended.

Session collaboration and derivatives

One opaque jcs_ capability is the editable session identity. The shared thin Cloudflare relay verifies that capability against App D1 before allocating a namespaced Durable Object, carries ephemeral awareness/Yjs frames, and uses the same signaling channel to negotiate direct WebRTC between matching peers. The relay is not a canonical session store.

An editable session can mint an independent jcm_ live mirror. The App resolves that mirror to its source internally and exposes only GET routes for views, snapshots, and intelligence. Responses never echo the source session capability; deleting the source cascades the mirror. The relay performs the same internal resolution so a mirror participant can be visible in the room without learning the editable link. Frozen derivatives and forks remain separate future modes; the shared UI action system already represents those modes because CareerVector implements them today.

Core Model

There is one shared public-ad boundary:

flowchart LR
  topdown[JobCache top-down distributed ingest]
  bottomup[CareerVector bottom-up workspace lookup/write]
  obs[Observation envelope]
  ad[URL x language ad_id]
  payload[shared sparse payload]
  cell[payload cells]
  evidence[evidence]
  chunks[chunks]
  embeddings[Embedding records]
  role[URL-family role]
  workspace[CareerVector private workspace job]

  topdown --> obs
  bottomup --> obs
  obs --> ad
  obs --> payload --> cell
  evidence --> obs
  evidence --> chunks
  chunks --> embeddings
  ad --> role
  ad --> workspace

The shared boundary is objective public ad data: one URL x language ad_id, a sparse payload map keyed by the shared payload vocabulary, and each field's cell plus optional evidence refs and observed_at. The current contract calls that submitted payload map fields. An Observation is the full submitted envelope for ad observations. It does not carry generic addressing fields for roles or workspace jobs.

Roles, ads, and workspace jobs form one sparse payload tree. Every level uses the same information-carrying payload vocabulary. Node metadata/state is allowed to differ by level: ids, URLs, language identity, workflow status, trust/churn, files, leases, and timestamps are not shared payload just because a UI can show them. CareerVector workspace jobs are private workspace data; they may look up or write safe public ad payload through the shared boundary, but the workspace job itself is not in the CrateDB observation envelope.

Evaluations are deliberately outside this v0 payload tree. They may later roll up from workspace jobs to ads or roles, but that requires an explicit reducer and trust model.

Evidence, chunks, and embeddings support current cells, search, QA, and replay, but they do not define another data model. Embeddings are Embedding records for stable inputs, not observations.

Object storage is cold evidence, not the business-critical record. Losing or replacing an object should hurt replay, inspection, or re-extraction, not shared ad identity, URL-family role identity, private workspace state, or accepted field/cell state in CrateDB. The durable contract expects an online S3-compatible store, not a local filesystem path.

Ingest Draft

The current ingestion design draft is ingest-architecture-draft.md. It captures the source-coverage reference strategy, the catalog/adapter/broker split, the shared field/cell payload model, and the efficiency rules for keeping JobCache cheap enough to operate.

The implemented broker path is broker-flow.md. It documents the operational chain from source/binding/task planning through device leasing, observation submission, current ad payload/state, device trust, and verification tasks.

The Observation model is observation.md. It documents how repeated observations, conflicts, device trust, observation freshness, and verification tasks produce current cells without making Elo or adapter code load-bearing.

The Adapter contract is adapter-contract.md. It defines the adapter vocabulary, output observation shape, forbidden boundaries, file ownership, validation commands, and acceptance criteria for adapter-only work.

The desktop edge runtime proposal is desktop-edge-runtime.md. It captures the Tauri/Svelte split, the hosted-web CORS boundary, and the opt-in native device shape for desktop contribution.

The distributed device contract is device-contract.md. It defines the task/observation boundary and parallel implementation split while leaving source-specific adapter code out of the architecture track.

The shared vocabulary is glossary.md. New code and docs should use those terms so source, adapter, device, session, task, observation, ad, field, cell, role, and workspace job each mean one thing.

CareerVector Integration Boundary

CareerVector and JobCache meet through the shared CrateDB hot store, not through a JobCache service handshake. The common contract is the shared ad identity, payload field definitions, cell payloads, current-value rules, and attachment keys. Product domains still diverge: JobCache owns top-down distributed ingest, coverage, device orchestration, and operational review; CareerVector owns workspace creation, user workflow, CVL/documents, scoring, and application tracking. The shared layer is data-level, not product-level.

At workspace job creation time, CareerVector normalizes the submitted URL, preserves the language hint, and looks up shared URL x language ad_id records in CrateDB. If the ad exists, the private workspace job can resolve safe shared ad cells. If it does not exist, CareerVector still creates the private workspace job immediately and may submit safe public ad Observations into the same submitObservation writer used by JobCache. For new upward writes, the stable rule is Observations at the shared boundary, not legacy wrappers or flat cell commands: RADAR should reuse the scrape Observation it already produced, while other system-owned job patches need an explicit Observation over the shared field vocabulary before they publish upward.

CareerVector-origin Observations are published through the D1 command outbox, not through best-effort service calls. The workspace commit that creates or updates the private job must also enqueue the matching jobcache_commands submit_observation row in the same D1 batch. JobCache claims and drains that command asynchronously, then applies the Observation to CrateDB through the shared writer. This avoids a cross-DB transaction while keeping the workspace commit and publication intent durable together. Future direct trusted HTTP ingestion would be a separate surface for trusted publishers; it is not the CareerVector /ops path for workspace-origin Observations.

JobCache's distributed top-down ingest and CareerVector's synchronous bottom-up workspace lookup/write therefore use the same objective URL x language ad_id/payload/cell model at the shared boundary. The boundary to protect is the DB schema, payload contract, and current-value rules. Above that, the products intentionally keep their own surfaces, workflows, and metadata/state.

Packaging — why split into shared/ + interface/ + ingest/?

The data layer (@cv/jobcache-shared) is imported by the hosted interface and by trusted device/local/operator harnesses. The traffic shapes are different: the hosted interface is request-driven control-plane work, while harnesses execute bounded tasks on devices or operator-owned machines. Keeping ingest/ separate prevents browser/rendering/fetch dependencies from entering the hosted interface container.

A shared/ package keeps the cross-service imports explicit (no relative ../../../ paths from one package into another) and lets the per-package tsconfigs differ without duplicating the data layer.

Why CrateDB instead of SQLite or Postgres?

The original hot DB was CockroachDB Serverless (~6 GB rent-free, horizontal scaling provisioned out of the box). Cockroach disabled the free RU tier, so the hot DB was cut over to CrateDB Cloud CRFREE on 2026-06-08 — see jobcache/migrations/cratedb/CRATEDB-CUTOVER.md. CrateDB speaks pg-wire but is not Postgres: no JSONB, no transactions, no foreign keys, so the query layer follows the dialect contract in jobcache/shared/src/pg-options.ts. Postgres would require provisioning a server explicitly. App-local state uses D1/SQLite semantics; the shared searchable corpus remains in CrateDB.

Where the data flows

  1. Top-down ingest: JobCache plans source work, leases tasks to devices, and applies valid observations to current ad cells and attachment keys.
  2. Bottom-up workspace lookup/write: CareerVector imports a workspace URL, creates private workspace state, and enqueues any safe public ad Observation as a D1 jobcache_commands row in the same workspace commit batch.
  3. Cell write: submitObservation writes sparse current cells, evidence keys, chunk keys, embedding keys, device trust, and follow-up tasks to CrateDB.
  4. Cold evidence: Large raw blobs can be written to an online S3-compatible store by stable key. They remain replaceable evidence, not the business record.
  5. Read surfaces: CareerVector UI/API/MCP, JobCache UI/API/MCP, search, QA, and ops read resolved cells from the shared boundary and private workspace projections where applicable.
Source: jobcache/wiki/content/architecture/README.md