Adapter Dock

What this is

The Adapter dock is the language-neutral Rust contract every job-source observer plugs into. One implementation per source (or per source produced by a multi-tenant family factory) discovers candidate URLs and turns one URL into one Observation. The dock itself does no fetching, no parsing, no I/O — every host capability (HTTP, rate-limit hint, user-agent, optional JS-rendered fetch) is injected through the Env trait.

This is the layer between [[jobcache-top-down-ingestion]] and the Observation write path documented in JOBCACHE-ARCHITECTURE.md. The shared trait and helpers live in crates/jobcache/; each adapter family lives in crates/jobcache-adapter-<family>/.

Why Rust, why cdylib + rlib

Every adapter crate is configured as both cdylib (the WASM Component Model artifact loaded by Node ingest, browsers, and sandboxed hosts) AND rlib (native, directly linked into the Tauri desktop runtime and into the crate's own unit tests). Same source, two targets, chosen at deploy time. There are no #[cfg(target_arch = "wasm32")] branches inside an adapter; every host capability comes in through the Env trait, so the adapter never needs to know whether it is running native or in a WASM component. From the file header at /home/richc/Documents/GitHub/careervector/crates/jobcache/src/adapter.rs:

This is what makes one Rust source crate compile to both a native rlib (Tauri desktop direct-link, with a Tauri-backed Env) and a WASM Component Model artifact (Node ingest, with a JS-fetch-backed Env) — no #[cfg(target_arch = "wasm32")] branches inside the adapter.

Relationship to lib/jobcache TypeScript mirror

crates/jobcache/ is the Rust mirror of @cv/jobcache on the TypeScript side (lib/jobcache/src/adapter.ts, contract.ts, parsed-ad.ts, ids.ts). Both sides are intentionally byte-equivalent for shared identity (url_id, ad_id_from_url_language, observation hash inputs), and the serde representations match the TS JSON shape (snake_case field keys, kebab-case enum variants for Method / EvidenceRefKind, lowercase for ObservationStatus / TaskClass). Drift here means ad_id divergence across the host boundary, which breaks dedup.

The Rust side is the surface every adapter implements. The TS mirror is the surface the Node ingest broker reads/writes after a WASM-loaded adapter has emitted an Observation.

The Adapter trait

Defined in /home/richc/Documents/GitHub/careervector/crates/jobcache/src/adapter.rs:

#[async_trait]
pub trait Adapter: Send + Sync {
    fn descriptor(&self) -> AdapterDescriptor;

    async fn discover(&self, env: &dyn Env) -> Result<Vec<String>, AdapterError>;

    async fn observe(
        &self,
        url: &str,
        env: &dyn Env,
        opts: &ObservationOptions,
    ) -> Result<Option<Observation>, AdapterError>;
}

Three methods, intentionally minimal:

• descriptor() — identity: id, version, label, homepage, method. Synchronous, infallible, must be cheap (the router calls it on every URL match attempt).
• discover(env) — return candidate URLs the source has on offer. Bounded — the adapter is expected to cap its own return (each crate defines MAX_DISCOVER or an equivalent) so the device runtime never receives unbounded lists.
• observe(url, env, opts) — fetch + parse one URL into one Observation. Return Ok(None) when the URL turned out to be irrelevant (expired, redirected to a non-job page, content missing). Return Err(AdapterError::Fetch | Parse | Contract | Other) for transport or parse failures the runtime should surface.

AdapterDescriptor

pub struct AdapterDescriptor {
    pub id: String,
    pub version: String,
    pub label: String,
    pub homepage: String,
    pub method: Method,
}

id is the stable adapter identifier (e.g. "lever:anybotics", "sf-spa:pictet") and persists into the Observation.adapter_id field. homepage is also used by Router::match_url for URL → adapter dispatch (see /home/richc/Documents/GitHub/careervector/ui/desktop/src-tauri/src/runtime/router.rs) — the router treats homepage as a prefix and tolerates http:// ↔ https:// and www. ↔ bare-host variants.

Method enum

pub enum Method {
    SitemapJsonld,
    HtmlBespoke,
    HeadlessProxy,
    FeedApi,
}

Serialized as kebab-case ("sitemap-jsonld", "html-bespoke", "headless-proxy", "feed-api"). It is surfaced on the dashboard AND used by the device runtime to decide which kind of Env.fetch implementation to inject (e.g. JS-capable proxy chain for HeadlessProxy).

Observation envelope and ParsedAd shape

The submitted shape is Observation (cross-reference JOBCACHE-ARCHITECTURE.md: "Observation is the canonical submitted envelope for public ad output. It is anchored to one ad_id and contains shared payload fields/cells plus evidence, chunks, embeddings, and provenance when produced"). Full type lives at /home/richc/Documents/GitHub/careervector/crates/jobcache/src/contract.rs, keyed off the 28-entry FieldKey vocabulary (title, organization, description, description_html, location, locality, region, country_code, remote, employment_type, language, industry, salary, salary_min, salary_max, salary_currency, salary_period, workload, source, source_id, source_ref, url, resolved_url, canonical_url, posted_at, valid_through, raw — see FieldKey::ALL). BTreeMap<FieldKey, ObservedField> ordering matches Object.keys().sort() on the TS side (lexicographic on the serialized name).

Adapters that produce a flat ad shape internally use the ParsedAd helper at /home/richc/Documents/GitHub/careervector/crates/jobcache/src/parsed_ad.rs, then call parsed_ad_to_observation(&ad, &opts) to wrap it into a full envelope (evidence refs, content hash, default resource policy, observation hash, fallback ids). ParsedAd is NOT a submitted shape — only Observation crosses the dock.

ObservationOptions — runtime envelope hints

Optional hints the device runtime passes into observe() to fill in envelope fields it owns (task/lease/device/session ids, app and runtime versions, status, timing, hashes). Adapters use these as defaults when constructing the Observation; the runtime can also fill them in afterwards. Defined in crates/jobcache/src/adapter.rs. All fields optional; ObservationOptions::default() serializes to {}. The serializer matches the TS dock — camelCase on the wire (taskId, leaseId, deviceId, adapterId, adapterVersion, fetchedAt, observedAt, timingMs, bytesRead, contentHash, adId, …).

Env trait — host-injected capabilities

#[async_trait]
pub trait Env: Send + Sync {
    fn user_agent(&self) -> &str;

    fn rate_limit(&self) -> Option<RateLimit> { None }

    async fn fetch(&self, req: Request) -> Result<Response, FetchError>;

    async fn fetch_rendered(&self, _req: Request) -> Result<Response, FetchError> {
        Err(FetchError::Unavailable("rendered fetch".into()))
    }
}

The adapter never constructs an Env; the device runtime (Tauri host, Node host, test harness, etc.) implements Env and passes a borrowed reference to the adapter on every discover() / observe() call.

• user_agent() — UA string the adapter should claim. Required.
• rate_limit() — optional RateLimit { per_minute } advisory the runtime is allowed to ignore.
• fetch(req) — plain HTTP fetch, no JavaScript execution. Used for sources whose listing/detail pages are server-rendered or expose a JSON API.
• fetch_rendered(req) — JavaScript-rendered fetch; drives a headless browser and returns the post-render DOM as HTML in response.body. Default impl returns FetchError::Unavailable — adapters that need this capability should propagate the error so the runtime can mark the source as "needs rendered fetch backend" and skip it cleanly. Hosts that wire a real backend (Browserbase, Firecrawl, local Playwright) override this method.

Request carries url, HTTP method (defaults to "GET"), header list, optional body, and timeout_ms (defaults to 30 000). Build via Request::get(url) for the common case. Response carries status, headers, raw body bytes, and final_url (which can differ from the requested URL after redirects).

Current Env implementations

• TauriEnv — native runtime backing for the Tauri desktop app (crates/jobcache/ consumers wire it through the desktop runtime router). fetch_rendered not yet wired — see "Future capabilities" below.
• MockEnv — test-only impl at /home/richc/Documents/GitHub/careervector/crates/jobcache/src/test_env.rs. Used by every adapter crate's unit tests.
• Node ingest runtime — future: a JS-fetch-backed Env that the WASM Component Model artifact will receive when loaded into jobcache/ingest.

FetchError taxonomy

pub enum FetchError {
    Timeout,
    Aborted,
    Network(String),
    RateLimited(u32),
    BadStatus(u16),
    Unavailable(String),
    RobotsBlocked(String),
}

Fetch policy

The Tauri desktop runtime's TauriEnv implements fetch and fetch_rendered with FOUR layers, in this order:

1. robots.txt (per-domain cached, hard pre-check). Disallowed → FetchError::RobotsBlocked. Per-request opt-out via Request::ignoring_robots() is reserved for workspace-owner-consenting calls.
2. Per-host rate limit (governor GCRA bucket, default 30 req/min). Concurrent requests against the same host serialize on the bucket.
3. Single attempt — reqwest::Client::send() (or the hidden WebviewWindow render for fetch_rendered). The result propagates directly to the caller UNLESS the request opted into retry.
4. Opt-in retry loop — when the adapter built the request via Request::get(url).with_retry(RetryPolicy { max_attempts, retry_on }), the single-attempt result feeds an exponential-backoff loop with ±25 % jitter. RateLimited honours the server's Retry-After.

Why one attempt by default

The default is ONE attempt + soft-fail because that's what years of jobich production (Swiss + EU job-board scraping, parser/scrapers.py in the jobich repo) say works. Hostile boards (LinkedIn, Indeed, Glassdoor) account-flag on retry storms; multiple 429s in close succession escalates to permanent bans. The polite-client shape — rate limit + robots + soft-fail — beats aggressive retry every time on this kind of source.

Per-source retry opt-in

A small set of adapters genuinely benefit from a bounded retry budget. Today these are:

Adapters opt in via:

let req = jobcache::Request::get(url).with_retry(RetryPolicy {
    max_attempts: 2,
    retry_on: vec![RetryClass::ServerError],
});
let resp = env.fetch(req).await?;

RetryClass::ServerError is the safe default opt-in — covers 502/503/504 without touching the throttle classes (429, RateLimited) that escalate to bans on hostile boards. Timeout / Network / RateLimited exist for adapters with a documented failure mode they protect against (specifically, a vendor-API tenant with measured intermittent network glitches that one retry catches). Don't reach for them speculatively.

AdapterError taxonomy

pub enum AdapterError {
    Fetch(FetchError),  // #[from] FetchError
    Parse(String),
    Contract(String),
    Other(String),
}

observe() returning Ok(None) is NOT an error: it means the URL turned out to be irrelevant (expired, redirected, content missing) and the runtime should drop it cleanly without raising a failure.

url_id contract (frozen)

crates/jobcache/src/url_id.rs is the byte-equivalent mirror of lib/jobcache/src/ids.ts. Immutable contract — changing the rules retroactively shifts every shared id and breaks dedup.

Public functions:

• normalize_url(input) -> Result<String, UrlIdError> — parse, lowercase host, strip fragment, strip a fixed list of tracking params (utm_*, fbclid, gclid, mc_eid, mc_cid, _ga, _gl, ref, referrer, source), sort remaining params, trim a single trailing / from non-root paths, decode unreserved percent-escapes (and uppercase the hex of reserved ones per RFC 3986 §6.2.2.1).
• hash_text(input) -> Result<String, UrlIdError> — SHA-256, truncate to 10 bytes, RFC 4648 base32 lowercase no padding. 16 chars.
• hash_url(url) -> Result<String, UrlIdError> — normalize then hash.
• ad_id_from_url_language(url, language) -> Result<String, UrlIdError> — hash_text("ad:{normalized}\nlanguage:{lang}"). Splits by URL × language; language=None becomes the "und" unknown-language bucket.
• role_id_from_url(url) -> Result<String, UrlIdError> — "role_{hash}" where the hash is hash_text("role:{normalized}"). Intentionally language-independent: ads split by URL × language, roles do not.
• job_id_from_url(url) -> Result<String, UrlIdError> — "job_{hash}" where the hash is hash_text("job:{normalized}"). The public workspace job id exposed in workspace APIs, routes, and Y.Doc keys. Intentionally NOT equal to hash_url(url) even though both run over the normalized URL.

Cross-language golden vectors are pinned in the test module at the bottom of url_id.rs; any drift between the Rust and TypeScript impls must keep those test vectors green.

SCHEMA_ID = "cv.jobcache"

pub const SCHEMA_ID: &str = "cv.jobcache";
pub const SCHEMA_VERSION: &str = "v0";

Defined in crates/jobcache/src/contract.rs. Every Observation carries both. Frozen contract identifier — bump SCHEMA_VERSION to "v1" (etc.) when an incompatible envelope shape ships.

Family catalog

All 33 currently-registered families. Tenant counts come from /home/richc/Documents/GitHub/jobich/parser/sources.yaml (verified 2026-05-30). "Fetch mode" indicates which Env method the listing path uses; detail pages may use a different mode (notes in the last column).

Wave 0 — original 19 (Rust port of the Python reference)

Wave 1 — 9 static-HTML expansions

Wave 2 — 5 JS-rendered (use env.fetch_rendered)

Wave 3 — 56 employer-bespoke + public boards + Yousty platform

Major public boards (11)

Swiss job boards (13)

Recruitment agencies (15)

Single-employer JSON APIs (10)

Specialized vertical boards (6)

Yousty platform (1 crate, 3 tenants)

The TS jobcache/ingest/src/portals/yousty.ts covers 3 Swiss regional boards through a shared sitemap+JSON-LD abstraction. Ported as one Rust crate:

custom-html extension (9 new tenants, total 16)

The existing custom-html crate's SOURCES table grew by 9 Swiss employer one-offs: sbb, axpo, alpiq, bkw, migros, securitas, cern, syzgroup, startupch. Each follows the same per-tenant selector pattern already documented for the 7 original tenants.

How to add a new family

1. Create the crate directory under crates/jobcache-adapter-<name>/. The workspace Cargo.toml already has a crates/* glob, so a new directory is picked up automatically — no edit to the workspace manifest is required.
2. Implement Adapter in src/lib.rs. Choose Method per the table above. If the source has more than one tenant, follow the pub struct <Family>Source { id, label, ... } + pub const SOURCES: &[<Family>Source] = &[...] pattern, and produce one <Family>Adapter per source. Bound discover() output (define MAX_DISCOVER or an equivalent cap).
3. Add unit tests with MockEnv:
   • For plain-fetch adapters, register canned bodies via MockEnv::with_response(url, body) / with_json_response(url, body) / with_status(url, code) / with_redirect(url, final_url, body).
   • For JS-rendered adapters, ALSO register with_rendered_response(url, body) — the fixture should be the HTML you'd see in the browser AFTER JavaScript has hydrated the SPA, not the raw server response.
4. Expose pub fn all_adapters() -> Vec<<Family>Adapter> — every crate ships this factory; the desktop router calls it at startup.
5. Wire into the desktop runtime at /home/richc/Documents/GitHub/careervector/ui/desktop/src-tauri/src/runtime/router.rs: add a path = ".../jobcache-adapter-<name>" dep to that crate's Cargo.toml and append one extend_with(&mut adapters, jobcache_adapter_<name>::all_adapters()); line in Router::default().
6. Mirror in jobcache/ingest/src/adapters/registry.ts when the Node ingest broker is wired to call the WASM Component Model artifact.

Testing pattern

MockEnv (at crates/jobcache/src/test_env.rs) is the shared test harness. Its builder methods all return Self so registrations chain:

let env = MockEnv::new()
    .with_user_agent("jobcache-test/0.1")
    .with_response("https://example.com/jobs", LISTING_HTML)
    .with_json_response("https://example.com/api/jobs/42", DETAIL_JSON)
    .with_redirect(
        "https://example.com/jobs/42",
        "https://example.com/jobs/42/final",
        DETAIL_HTML,
    )
    .with_status("https://example.com/jobs/oops", 404)
    .with_rendered_response("https://spa.example/career", POST_HYDRATION_HTML);

let urls = adapter.discover(&env).await.unwrap();
let obs = adapter.observe(&urls[0], &env, &ObservationOptions::default())
    .await
    .unwrap();

env.calls() and env.rendered_calls() return every URL each fetch channel was invoked with, in order, so tests can assert e.g. that the adapter prefers plain fetch and only falls back to fetch_rendered when needed.

Worked examples to read before writing a new family:

• crates/jobcache-adapter-lever/src/lib.rs — canonical JSON-API (Method::FeedApi) reference.
• crates/jobcache-adapter-icims/src/lib.rs — canonical HTML + schema.org reference (delegates observe to parse_schema_org_jobposting).
• crates/jobcache-adapter-sf-spa/src/lib.rs — canonical Method::HeadlessProxy reference, including the plain-first / rendered- fallback pattern for detail pages with optional JSON-LD.
• crates/jobcache-adapter-custom-html/src/lib.rs + crates/jobcache-adapter-playwright-html/src/lib.rs — per-tenant selector pattern (data-driven, one Adapter impl, N tenants).
• crates/jobcache-adapter-hibob/src/lib.rs — the "found a JSON endpoint" exception. Read before assuming a JS-rendered front-end forces Method::HeadlessProxy.

Future capabilities

• TauriEnv::fetch_rendered backend wiring — pending. Candidate backends: Browserbase, Firecrawl, local Playwright. Until one is wired, every HeadlessProxy adapter surfaces FetchError::Unavailable on a desktop install and the runtime skips it.
• Node ingest runtime providing Env — the WASM Component Model loader inside jobcache/ingest will pass in a JS-fetch-backed Env so every adapter that compiles to cdylib runs there as well.
• Multi-step rendered interactions — fetch_rendered returns one post-hydration page. Adapters that would benefit from "render, then click N times" (e.g. SF SPA pagination, CSOD pagination) currently harvest only the first rendered page. Extending the dock with a multi-step interaction primitive would unlock those tails; today it is intentionally out of scope.