Ingest Architecture Draft

JobCache is the top-down ingest trigger for CareerVector's job knowledge graph. It should learn aggressively from proven Swiss job aggregation systems, especially their scraping process and source coverage, while keeping the implementation, storage model, and product boundary native to CareerVector.

North Star

Build a continuously refreshed CH/EU shared ad set that is:

  • broad enough to be useful before a user imports a job manually
  • cheap enough to run on the free/low-cost stack
  • explainable enough that every field can be traced back to a source snapshot
  • structured enough to improve CareerVector matching, evaluation, and job discovery
  • isolated enough that workspace-private CVs, notes, files, conversations, and UI state never leak into the shared ad data layer

The shared product boundary remains:

ad_id -> fields[field].cell

An Observation anchors to one ad_id. Its fields object is keyed by shared field vocabulary, and each field contains a cell plus optional evidence refs and observed_at. The ad identity is URL x language via adIdFromUrlLanguage(url, language). The role identity above language-specific ads is URL-family identity via roleIdFromUrl(url). Workspace jobs are private CareerVector records; they may look up or write safe public ad fields, but they are not part of the submitted observation envelope.

There are two directions, but only one tree:

top-down ingest:   JobCache discovers ads -> ad cells and attachments
bottom-up write:   CareerVector imports a URL -> private workspace job plus safe public ad fields
read surfaces:     CareerVector UI/API/MCP, JobCache UI/API/MCP, search, QA, ops

JobCache and CareerVector meet at the same objective ad/field/cell boundary, not through separate product data shapes. A workspace URL import should not stay trapped inside workspace state forever. After workspace-private fields are stripped, safe public observations from the bottom-up path should pass through the same identity, provenance, device-trust, freshness, deduplication, and submitObservation rules as top-down JobCache ingest.

Workspace-private CVL, notes, files, conversations, user evaluations, and UI state stay below the shared boundary. Public ad evidence, source URL, normalized text, content hashes, observed fields/cells, chunks, embeddings, and role-link metadata can move upward when they satisfy the shared boundary submission rules.

The common submitted shape is the @cv/jobcache Observation. JobCache top-down discovery and CareerVector bottom-up URL imports both emit and consume that same ad_id-centered envelope after workspace-private fields are stripped. submitObservation is the coordination point: it validates schema, lease, and policy; writes current ad payload cells; persists emitted object_refs, chunk_refs, annotations, search_chunks, and embeddings as hot index rows; updates global device trust; and queues verify-observation tasks only on fresh conflict. It does not fetch, chunk, enrich, embed, or run LLMs on the server. CareerVector does a synchronous lookup in the shared ad tables when a workspace job URL is created. JobCache's top-down devices write into the same shared boundary asynchronously.

CareerVector bottom-up single-URL import calls scrapeUrl(). The current path runs the ordered browser-compatible adapter/provider chain, including Jina/provider-chain readout. When an Adapter runs on a Device, that Adapter plus Device is the Scraper. RADAR is the CareerVector workspace entry surface that can search the shared database, accept a manually added URL, or surface machine-made candidates. Raw source/API/provider output stays inside adapter internals. Observation is the canonical submitted task output; DB writes, trust policy, queue scheduling, and workspace data stay outside adapter code. Operational HTTP surfaces may exist for dashboards, MCP, health, or admin work, but the CareerVector/JobCache product contract is shared ad identity, field/cell payload shape, attachment keys, and submitObservation rules.

Reference Scope

External job aggregators are useful references for source coverage and operating behaviour. They are not references for the business data shape. JobCache should use them to understand which ATS families, source types, parsing stages, stale-checks, source-health controls, and search-quality loops matter, then express those behaviours through CareerVector's tree and cell model.

The operating guideline is dark grey, not black:

  • dark grey: study source coverage, workflow shape, reliability tactics, stale checks, dedup behaviour, and operational loops closely enough that JobCache competes with proven systems
  • black: import another product's code, prose, fixtures, arbitrary constants, source registry, or flat vacancy data model as canonical JobCache material

Reference Behaviours

Job aggregation gets good through coverage and operational iteration. The useful reference behaviours are:

  • broad ATS family coverage: Workday, Greenhouse, Lever, SuccessFactors, SmartRecruiters, Personio, Teamtailor, Ashby, Recruitee, BambooHR, HiBob, Workable, Oracle HCM, Refline, Prospective, Phenom, RSS, custom HTML, and JS-rendered career portals
  • source config ergonomics: adding a company on a supported family should be mostly a registry entry, not bespoke application work
  • daily and targeted source runs: full sweeps for broad freshness, narrow reruns for repair and source debugging
  • extraction stages: listing discovery, detail fetch, field extraction, normalization, enrichment, identity/fingerprint, persistence, and stale checks
  • cleanup/enrichment fields: canton/region, language, industry, seniority, employment type, work mode, salary band, skills, source, first seen, last seen, and validity window
  • search quality loops: FTS, semantic embeddings, duplicate handling, facets, and operator-visible bad-result reports
  • operational guardrails: source status, pause/rerun actions, kill switches, error categories, quotas, and cost per successful posting

What JobCache Should Do Differently

JobCache must be more efficient than app-centric aggregators.

  • Store bulky raw material as replaceable online S3-compatible evidence, not hot SQL rows.
  • Store only structured queryable cells, hashes, keys, metrics, and embeddings in CockroachDB.
  • Treat Cockroach tree identities, field/cell payloads, attachment keys, and current cell provenance as the business record.
  • Use content fingerprints so unchanged observed rows do not trigger repeated writes, enrichment, embeddings, or deferred reduction tasks.
  • Separate catalog, adapter, and broker so source knowledge, execution mechanics, and orchestration can evolve independently.
  • Avoid full-data-set jobs unless the operator intentionally starts one.
  • Prefer incremental source windows, source-specific freshness budgets, and bounded batches.
  • Treat every expensive operation as cacheable by stable input key.
  • Make every scrape attempt observable so efficiency regressions are visible before they become a bill.

Pipeline

source registry
  -> adapter.discover()
  -> adapter.fetch()
  -> adapter.parse()
  -> Observation
  -> submitObservation validates schema/lease/policy
  -> current ad payload cells + emitted object/chunk/annotation/search/embedding indexes
  -> global device trust and verify-observation tasks on fresh conflict
  -> URL-family role links and read surfaces
  -> scrape event + metrics

This draft intentionally defines the contract around ingestion, not adapter implementation. Source adapters can be written independently as long as they return the agreed Observation shape and never bypass submitObservation.

Source And Orchestration Model

Do not model this as "one adapter per job site." That becomes too bespoke too quickly.

Model it as:

family         reusable job-site type, e.g. Workday, Greenhouse, jobs.ch, custom-html
source         concrete job site or tenant, e.g. UBS Workday, jobs.ch CH
adapter        versioned code that handles one family or exact source
binding        config connecting one source to one adapter
task           small planned unit: discover, fetch_detail, parse, embed, verify
lease          temporary assignment of one task to one device
Observation    task output submitted through submitObservation
fields/cells   current compact shared state in Cockroach

The usual case should be many source rows sharing one adapter. A bespoke source gets its own adapter only when no family helper can describe it.

Examples:

Workday adapter
  -> source: UBS careers
  -> source: Roche careers
  -> source: Swisscom careers

jobs.ch adapter
  -> source: jobs.ch CH

custom-html adapter
  -> source: small company career page

This is also the orchestration model. The planner reads sources and bindings, decides what work is due, creates small task rows, and the broker leases those tasks to available devices. Devices do not decide the crawl frontier; they only execute leased tasks.

Minimum conceptual records:

  • families: reusable source/platform type.
  • sources: concrete job site, board, tenant, or company career page.
  • adapters: versioned implementation that can execute tasks for one or more families.
  • bindings: source plus adapter plus config, freshness, risk, and task limits.
  • tasks: planned units of work with kind, source, URL/seed, adapter version, state, priority, and stable input key.
  • leases: short-lived device assignment for one task.
  • devices: enrolled machines, capabilities, resource policy, device trust, and last heartbeat.
  • sessions: running app instances/processes on devices.
  • current ad payload cells: current public field/cell values for an ad_id, with provenance and freshness metadata. The current implementation table is named ad_facts.
  • ads: shared public ad identity, keyed by URL x language.
  • roles: URL-family role nodes above one or more language-specific ads.
  • workspace jobs: private CareerVector state that can look up or write safe public ad fields.
  • fields, cells: shared payload infrastructure for local values and resolved projections.
  • evidence, object_refs, text_chunks, ad_annotations, embeddings: attached support data keyed to ads, fields, chunks, and stable input hashes.

The DB is the coordination point. Planner and submitObservation code can be small and central; expensive work is leased to clients.

Source Catalog

The catalog describes the family, source, adapter, and binding:

  • source id and display name
  • ATS/platform family
  • seed URLs or API endpoint shape
  • country/location filters
  • expected selectors or JSON paths
  • fetch requirements such as JS rendering
  • freshness target and batch limits
  • risk/cost class for operator visibility

The catalog is not the ad/role data model. It is the operating catalog that tells ingest what to try.

Adapter Semantics

An adapter owns source semantics:

  • discovery URL shape
  • sitemap/API/listing pagination
  • detail URL extraction
  • JSON-LD or source-specific field mapping
  • source-specific validity checks
  • stale/closed markers when known

Adapter logic should be small and boring. Source complexity should be pushed into reusable family helpers where possible. The important design goal is that adding a new supported-family source is mostly inserting a source and binding, not writing a new adapter.

Adapter Execution

An adapter owns execution mechanics:

  • raw HTTP fetch
  • managed JS rendering
  • browser-backed rendering
  • future externally supplied fetch executors

The rest of the pipeline should not care how HTML or JSON was obtained. It receives a fetch result plus provenance, timings, and error classification.

Adapter execution is deliberately swappable. The architecture owner defines the task/observation contract, provenance requirements, resource policy, global device-trust updates, DB writes, and validation. Adapter implementers own source-specific fetch and parse code behind that contract.

The preferred implementation split is:

  • Rust native device runtime for desktop execution, queueing, resource checks, hashing, compression, and model-pack execution where needed.
  • Svelte/TypeScript UI and orchestration surfaces.
  • JSON-compatible task/observation messages as the stable boundary between broker, device, and Adapters.
  • Optional TypeScript adapter modules only for pure parsing/normalization logic where that is simpler than Rust.

No adapter should get Cockroach credentials. Devices submit Observations through submitObservation; only server-side DB code commits current ad payload cells, attachments, and provenance. That DB code persists emitted support records; it does not call fetchers, chunkers, enrichers, embedding models, or LLMs.

Broker

The broker chooses which planned tasks to lease and which device is allowed to run them. It records every attempt.

Core behaviours:

  • cheapest-first where more than one execution path is available
  • per-host spacing
  • 429 cooldown
  • executor quota gates
  • bounded retries
  • structured error reasons
  • source-level cost and latency metrics

The broker is not a source-specific hack table. It is a policy layer for cost, capability, trust, device availability, and failure handling.

The concrete broker path is tracked in broker-flow.md: source and binding planning creates tasks; devices lease tasks; devices submit Observations; submitObservation writes current ad payload cells, persists emitted support indexes, updates global device trust, and queues verify-observation tasks on fresh conflict.

Normalize And Enrich

Normalization should be deterministic first:

  • canonical URL and resolved URL
  • title normalization
  • company name normalization
  • location/canton/region
  • country code
  • language
  • employment type and work mode
  • salary min/max/currency/period
  • seniority
  • skills and technology tags
  • source family and source id

LLM or embedding-based enrichment should only run when deterministic extraction is insufficient and the input hash has changed.

Identity And Dedup

JobCache should not flatten everything into one vacancy row.

Use the shared tree identity:

  • role: URL-family node for one canonical URL family, with later async reduction able to connect wider semantic roles outside the hot path
  • ad: canonical public listing language under a role
  • workspace job: private CareerVector record linked to public ad data

Observed source rows, detail fetches, and imported pages attach as observations and evidence under the relevant URL x language ad_id or candidate ad. They are not a fourth business node. Role identity is derived from URL family with roleIdFromUrl(url); cross-URL semantic role reduction runs later as async database work.

Potential matching evidence:

  • canonical URL
  • source job id
  • company normalized name
  • title normalized form
  • location
  • description fingerprint
  • embedding similarity
  • salary/workload/employment-type compatibility
  • posting lifecycle overlap

Disagreement is represented as Observations, current cells, and provenance. The last valid Observation wins the current ad payload, while lightweight state and evidence keep the disagreement traceable.

Storage Split

CockroachDB is the hot structured store:

  • ads, URL-family roles, and workspace lookup links
  • field definitions and cells
  • current ad payload cells
  • edges
  • evidence keys and metadata
  • object refs, chunk metadata, annotations, and stable input keys
  • source registry state
  • scrape events
  • operator actions
  • fingerprints
  • embeddings
  • search/facet materializations

Online S3-compatible object storage is the replaceable cold evidence store. B2 is the current default; R2, S3, MinIO, or Garage can replace it if they remain addressable from devices and later replay or QA tooling. submitObservation validates submitted object references; it does not read object-store blobs.

  • raw HTML
  • raw JSON/API responses
  • JSON-LD payloads
  • extraction outputs
  • evidence bundles for QA and operator review

The hot row stores the content hash, stable input key, and optional object key. The blob stores raw evidence that can be re-fetched, regenerated, or discarded without changing the business identity or current cells. Losing an object store hurts replay and inspection; Cockroach remains the system of record.

Retrieval And Findability

Jobich demonstrates that a good shared ad set is not enough. Findability and precision need a first-class architecture.

Baseline retrieval should combine:

  • structured filters
  • Cockroach full-text search
  • vector search for semantic recall
  • graph/tree expansion around candidate roles, ads, and workspace jobs
  • explicit ranking features from resolved cells

Future RAG work should use the job tree and its attachments as product data, not as an imported framework. The tree should help answer:

  • why this job matches a workspace
  • which ads are likely the same role
  • why two postings disagree
  • what changed since the last scrape
  • which companies/sources dominate a market slice
  • where source coverage is weak

This is a later design topic. The ingest pipeline should preserve enough evidence and structure that graph-aware retrieval and RAG can be added without re-scraping the world.

Efficiency Rules

Efficiency is a product requirement, not an optimization pass.

  • Never re-embed unchanged text.
  • Never re-enrich unchanged normalized fields.
  • Never update a row just to refresh identical data.
  • Never store a large blob in hot SQL when a hash plus replaceable evidence key is enough.
  • Prefer append-only event rows for attempts and operator actions.
  • Keep source runs bounded by explicit max and freshness budgets.
  • Prefer source-specific deltas over full sweeps.
  • Materialize expensive facets and search helpers when query volume justifies it.
  • Use Cockroach EXPLAIN ANALYZE on representative queries before adding broad search/ranking features.
  • Keep large snapshots and extraction outputs out of Cockroach.

Open Decisions

  • Which source families become first-class helpers first.
  • Exact storage policy for embeddings after submitObservation. In the submitted payload they ride inside an Observation and reference stable chunks.
  • How much source registry state belongs in code versus Cockroach.
  • Exact async role-reduction trust thresholds and operator review flow.
  • How graph-aware retrieval should be exposed to CareerVector, MCP, and the jobcache UI.
  • Whether a separate graph database is ever justified, or whether Cockroach edge tables are enough for the first product year.
Source: jobcache/wiki/content/architecture/ingest-architecture-draft.md