Rust + WASM Adapter Architecture

End-state proposal for the JobCache Adapter dock. Written for Codex review — Codex owns the contract crate, WIT source, host shim, and build pipeline. Adapter implementer (me) owns the adapter crates that target this infrastructure.

Decisions baked in

Open decisions (call out below): exact async model in the trait, WIT versioning policy, Component-Model vs WASI-preview baseline, WASM crate size budget, CI/build pipeline particulars.

Goal: write once, use multiple times

One Rust source crate per family. The same crate compiles to multiple deployment artifacts:

Cargo layout per family crate:

[package]
name = "jobcache-adapter-jobs-ch"
version = "0.1.0"
edition = "2024"

[lib]
crate-type = ["cdylib", "rlib"]

[dependencies]
jobcache = { path = "../jobcache" }    # contract crate (Rust mirror of @cv/jobcache)
scraper = "0.20"                                      # html5ever-based HTML
serde_json = "1"
regex = "1"
url = "2"
sha2 = "0.10"
base32 = "0.5"
# wit-bindgen pulled in only for the WASM target via [target.'cfg(target_arch = "wasm32")'.dependencies]

The dual-target contract

The contract has two surfaces that resolve to the same data shape:

1. Rust trait surface for native callers (Tauri direct link, native ingest if we ever go there, tests).
2. WIT/Component Model surface for WASM callers (Node ingest today, browser tomorrow, sandboxed contexts).

Both are generated from one source of truth. Two viable layouts:

• WIT is the source of truth. wit-bindgen generates both the WASM ABI bindings AND the Rust trait (a wit-bindgen host can also emit trait stubs that match the WIT). Tooling supports this with wit-bindgen rust for the WASM side and a manually-maintained Rust trait that the WASM bindings adapt to.
• Rust trait is the source of truth. Hand-write the trait and the WIT in parallel; CI checks they stay in sync. More maintenance, more readable Rust.

I'd recommend WIT as source of truth for future-proofing (polyglot adapter authors get the same contract for free), with the Rust trait derived. The contract crate jobcache exposes both.

WIT sketch (Codex-owned)

lib/jobcache/wit/jobcache-adapter.wit (or wherever Codex prefers):

package jobcache:adapter@0.1.0;

interface fetcher {
    record request {
        url: string,
        method: string,        // "GET" / "POST" / ...
        headers: list<tuple<string, string>>,
        body: option<list<u8>>,
        timeout-ms: u32,
    }

    record response {
        status: u16,
        headers: list<tuple<string, string>>,
        body: list<u8>,
        final-url: string,
    }

    variant fetch-error {
        timeout,
        aborted,
        network(string),
        rate-limited(u32),     // retry-after seconds
        bad-status(u16),
    }

    fetch: func(req: request) -> result<response, fetch-error>;
}

interface env {
    use fetcher.{request, response, fetch-error};
    user-agent: func() -> string;
    rate-limit-per-minute: func() -> option<u32>;
    fetch: func(req: request) -> result<response, fetch-error>;
}

interface observation-types {
    record evidence-ref {
        evidence-ref-id: string,
        kind: string,                                 // matches EvidenceRefKindSchema
        url: option<string>,
        content-hash: option<string>,
        observation-hash: option<string>,
        object-ref-id: option<string>,
        chunk-id: option<string>,
        byte-start: option<u32>,
        byte-end: option<u32>,
        char-start: option<u32>,
        char-end: option<u32>,
        selector: option<string>,
        excerpt-hash: option<string>,
    }

    variant cell-payload {
        value(value-payload),
        unknown(option<string>),
        empty(option<string>),
        redacted(redacted-payload),
    }
    record value-payload { value: string, value-hash: option<string> }
    record redacted-payload { reason: option<string>, policy-id: option<string> }

    record observed-field {
        cell: cell-payload,
        evidence-ref-ids: list<string>,
        observed-at: option<string>,
    }

    record task-error {
        code: string,
        message: string,
        retryable: option<bool>,
        detail: option<string>,                       // JSON-encoded
    }

    record observation {
        schema-id: string,                            // "cv.jobcache"
        schema-version: string,                       // "v0"
        task-id: string,
        lease-id: option<string>,
        ad-id: string,
        device-id: string,
        session-id: string,
        app-version: string,
        device-runtime-version: string,
        adapter-id: string,
        adapter-version: string,
        status: string,                               // "success" | "partial" | "failed"
        fetched-at: string,
        observed-at: option<string>,
        url: string,
        resolved-url: string,
        url-hash: string,
        content-hash: string,
        observation-hash: string,
        timing-ms: u32,
        bytes-read: u32,
        resource-policy: string,                      // JSON-encoded snapshot
        fields: list<tuple<string, observed-field>>,
        evidence-refs: list<evidence-ref>,
        object-refs: option<string>,                  // JSON-encoded array
        chunk-refs: option<string>,
        annotations: option<string>,
        search-chunks: option<string>,
        embeddings: option<string>,
        errors: list<task-error>,
    }
}

interface adapter {
    use env.{env-handle: env};
    use observation-types.{observation};

    record observation-options {
        task-id: option<string>,
        lease-id: option<string>,
        device-id: option<string>,
        session-id: option<string>,
        app-version: option<string>,
        device-runtime-version: option<string>,
        adapter-id: option<string>,
        adapter-version: option<string>,
        fetched-at: option<string>,
        observed-at: option<string>,
        content-hash: option<string>,
        ad-id: option<string>,
    }

    record adapter-descriptor {
        id: string,
        version: string,
        label: string,
        homepage: string,
        method: string,                               // "sitemap-jsonld" | "html-bespoke" | "headless-proxy" | "feed-api"
    }

    descriptor: func() -> adapter-descriptor;
    discover: func(env: borrow<env>) -> result<list<string>, string>;
    observe: func(url: string, env: borrow<env>, opts: observation-options) -> result<option<observation>, string>;
}

world jobcache-adapter {
    import env;
    export adapter;
}

Notes on this sketch:

• Complex nested types (object refs, chunk refs, annotations, search chunks, embeddings, resource-policy) are serialized as JSON strings at the boundary to keep the WIT manageable; the contract crate parses them into typed Rust structs internally. If WIT proves expressive enough as it matures, these can be promoted to first-class records later — backwards-compatible since they're optional.
• Async: Component Model has future<T> / stream<T> in newer revisions. Until tooling is stable, the fetch import is synchronous from the WASM side and the host implements it as blocking-on-async (acceptable: hosts already serialize per-source work behind rate limits). Alternative path: async-bindgen + WASI Preview 2 futures. Decision point for Codex.
• Env is borrow<env> so the adapter doesn't own the host's runtime state; the host passes a resource handle and reclaims it after the call.

Rust contract crate (Codex-owned)

lib/jobcache/crates/jobcache/ (or top-level crates/). Mirrors the TypeScript types from lib/jobcache/src/contract.ts and the dock from lib/jobcache/src/adapter.ts.

// crates/jobcache/src/lib.rs (sketch)

pub mod contract;
pub mod adapter;
pub mod url_id;

pub use contract::*;
pub use adapter::*;
pub use url_id::*;

// crates/jobcache/src/contract.rs (sketch)

use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;

pub const SCHEMA_ID: &str = "cv.jobcache";
pub const SCHEMA_VERSION: &str = "v0";

#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]
#[serde(tag = "kind", rename_all = "kebab-case")]
pub enum CellPayload {
    Value { value: serde_json::Value, value_hash: Option<String> },
    Unknown { reason: Option<String> },
    Empty { reason: Option<String> },
    Redacted { reason: Option<String>, policy_id: Option<String> },
}

#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObservedField {
    pub cell: CellPayload,
    pub evidence_ref_ids: Vec<String>,
    pub observed_at: Option<String>,
}

#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]
pub struct EvidenceRef {
    pub evidence_ref_id: String,
    pub kind: EvidenceRefKind,
    pub url: Option<String>,
    pub content_hash: Option<String>,
    pub observation_hash: Option<String>,
    pub object_ref_id: Option<String>,
    pub chunk_id: Option<String>,
    pub byte_start: Option<u32>,
    pub byte_end: Option<u32>,
    pub char_start: Option<u32>,
    pub char_end: Option<u32>,
    pub selector: Option<String>,
    pub excerpt_hash: Option<String>,
}

#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]
#[serde(rename_all = "kebab-case")]
pub enum EvidenceRefKind {
    SourceUrl,
    ContentHash,
    ObservationHash,
    TextRange,
    Html,
    Json,
    Screenshot,
    Object,
    Chunk,
}

#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]
pub struct Observation {
    pub schema_id: String,
    pub schema_version: String,
    pub task_id: String,
    pub lease_id: Option<String>,
    pub ad_id: String,
    pub device_id: String,
    pub session_id: String,
    pub app_version: String,
    pub device_runtime_version: String,
    pub adapter_id: String,
    pub adapter_version: String,
    pub status: ObservationStatus,
    pub fetched_at: String,
    pub observed_at: Option<String>,
    pub url: String,
    pub resolved_url: String,
    pub url_hash: String,
    pub content_hash: String,
    pub observation_hash: String,
    pub timing_ms: u64,
    pub bytes_read: u64,
    pub resource_policy: ResourcePolicySnapshot,
    pub fields: BTreeMap<FieldKey, ObservedField>,
    pub evidence_refs: Vec<EvidenceRef>,
    pub object_refs: Option<Vec<ObjectRef>>,
    pub chunk_refs: Option<Vec<ChunkRef>>,
    pub annotations: Option<Vec<Annotation>>,
    pub search_chunks: Option<Vec<ChunkRef>>,
    pub embeddings: Option<Vec<Embedding>>,
    pub errors: Vec<TaskError>,
}

// FieldKey is a strongly-typed enum mirroring FIELD_VOCABULARY exactly.
#[derive(Clone, Copy, Debug, Serialize, Deserialize, Eq, PartialEq, Ord, PartialOrd, Hash)]
#[serde(rename_all = "snake_case")]
pub enum FieldKey {
    Title, Organization, OrganizationUrl, Description, DescriptionHtml,
    Location, Locality, Region, CountryCode, Remote, EmploymentType,
    Language, Industry, Salary, SalaryMin, SalaryMax, SalaryCurrency,
    SalaryPeriod, Workload, Source, SourceId, SourceRef,
    Url, ResolvedUrl, CanonicalUrl, PostedAt, ValidThrough, Raw,
}

// ... ObjectRef, ChunkRef, Annotation, Embedding, TaskError, ResourcePolicySnapshot mirror contract.ts.

// crates/jobcache/src/adapter.rs (sketch)

use async_trait::async_trait;
use crate::contract::{Observation};

#[derive(Clone, Debug)]
pub struct AdapterDescriptor {
    pub id: String,
    pub version: String,
    pub label: String,
    pub homepage: String,
    pub method: Method,
}

#[derive(Clone, Copy, Debug)]
pub enum Method {
    SitemapJsonld,
    HtmlBespoke,
    HeadlessProxy,
    FeedApi,
}

#[async_trait]
pub trait Env: Send + Sync {
    fn user_agent(&self) -> &str;
    fn rate_limit_per_minute(&self) -> Option<u32>;
    async fn fetch(&self, req: Request) -> Result<Response, FetchError>;
}

#[derive(Clone, Debug, Default)]
pub struct ObservationOptions {
    pub task_id: Option<String>,
    pub lease_id: Option<String>,
    pub device_id: Option<String>,
    pub session_id: Option<String>,
    pub app_version: Option<String>,
    pub device_runtime_version: Option<String>,
    pub adapter_id: Option<String>,
    pub adapter_version: Option<String>,
    pub fetched_at: Option<String>,
    pub observed_at: Option<String>,
    pub content_hash: Option<String>,
    pub ad_id: Option<String>,
}

#[async_trait]
pub trait Adapter: Send + Sync {
    fn descriptor(&self) -> AdapterDescriptor;
    async fn discover(&self, env: &dyn Env) -> Result<Vec<String>, AdapterError>;
    async fn observe(
        &self,
        url: &str,
        env: &dyn Env,
        opts: &ObservationOptions,
    ) -> Result<Option<Observation>, AdapterError>;
}

The async-trait choice keeps the API ergonomic on the native side. For the WASM target, the bindings layer adapts between the async trait and the Component Model's sync fetch (or future async, whichever lands).

TS host shim (Codex-owned)

@cv/jobcache/wasm-host.ts:

import { ComponentLoader } from "@bytecodealliance/jco";
import type { Adapter, Env, ObservationOptions } from "./adapter";
import type { Observation } from "./contract";

export interface WasmAdapterSource {
  /** Path to the .wasm file, or a fetched ArrayBuffer. */
  wasmBytes: ArrayBuffer | Uint8Array;
}

/**
 * Load a Rust adapter compiled to a Component Model WASM, return a TS
 * Adapter that registry.ts and ingest.ts can use unchanged.
 */
export async function loadWasmAdapter(src: WasmAdapterSource): Promise<Adapter> {
  const component = await ComponentLoader.load(src.wasmBytes);
  // Wire the host's Env (env.fetch, env.userAgent, etc.) into the component's imports.
  const instance = await component.instantiate({
    env: makeEnvImport(/* env passed at call time */),
  });
  return {
    ...instance.descriptor(),
    discover: async (env: Env) => instance.discover(env),
    observe: async (url, env, opts?: ObservationOptions) => instance.observe(url, env, opts ?? {}),
  };
}

The shim hides the WASM boundary from registry.ts. A registry entry is either a native TS Adapter (the four wrapped portals, while they live) or await loadWasmAdapter({ wasmBytes: jobsChWasm }) — same return type.

Build pipeline (joint fence)

Cargo workspace at repo root (or under jobcache/):

# Cargo.toml at repo root
[workspace]
resolver = "2"
members = [
    "crates/jobcache",
    "crates/jobcache-adapter-jobs-ch",
    "crates/jobcache-adapter-greenhouse",
    "crates/jobcache-adapter-smartrecruiters",
    # ... one crate per family as they land
]

[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
strip = "symbols"

Per-family build:

# WASM build (for Node ingest, browser, sandboxed hosts)
cargo build -p jobcache-adapter-jobs-ch --target wasm32-wasip1 --release
wasm-tools component new target/wasm32-wasip1/release/jobcache_adapter_jobs_ch.wasm \
    --adapt wasi_snapshot_preview1=adapters/wasi_snapshot_preview1.reactor.wasm \
    -o dist/jobs-ch.component.wasm
wasm-opt -O3 dist/jobs-ch.component.wasm -o dist/jobs-ch.component.wasm

# Native build (for Tauri direct link, tests)
cargo build -p jobcache-adapter-jobs-ch --release

Distribution: built .component.wasm files land in lib/jobcache/wasm/ (or wherever the host shim expects them). Node ingest reads them at startup. Tauri future build links the rlib directly in its Cargo workspace.

CI:

• cargo test --workspace — Rust unit tests, fast.
• cargo build --workspace --target wasm32-wasip1 — proves WASM portability.
• TS integration test: load each .component.wasm via the host shim, run discover + observe against frozen fixtures, validate the Observation through ObservationSchema.

Crate-size budget

Each family WASM .component.wasm should aim for under 1 MB after wasm-opt -O3. With scraper + serde_json + regex + url, expected size is ~500-800 KB per crate. Significantly larger means we're pulling in something we don't need (full tokio runtime, etc.).

Justification: 5 hot families × 1 MB = 5 MB total at the host loaded at startup. Acceptable for Render ingest; reasonable for Tauri (though Tauri uses native, so the WASM budget only constrains Node ingest + future browser path).

Existing TypeScript cleanup

No code stays around "for migration" once its Rust replacement ships. No parallel TS+WASM path for the same source.

Slice 0 — Infrastructure landing (Codex-led, with my contribution)

Codex implements:

1. Cargo workspace at repo root.
2. crates/jobcache/ — Rust mirror of @cv/jobcache types + adapter trait + URL/language identity helpers (byte-equivalent to TS).
3. lib/jobcache/wit/jobcache-adapter.wit — WIT contract.
4. lib/jobcache/src/wasm-host.ts — TS host shim using jco.
5. Build pipeline scripts + CI integration (cargo build + wasm-opt + WASM smoke test).
6. One adapter crate scaffold (crates/jobcache-adapter-jobs-ch/) as a working example.

I contribute:

• Port the jobs-ch adapter logic from adapters/jobs-ch.ts to Rust inside that scaffold.
• Fixture conversion (HTML fixture stays HTML; expected-Observation fixture in JSON).
• Rust-side parser tests + WASM-loaded integration test through the TS host shim.

Acceptance:

• jobs-ch WASM loads via loadWasmAdapter, registered in registry.ts.
• discover() and observe() produce byte-identical Observations to the TS version on the same fixtures.
• adapters/jobs-ch.ts deleted in the same PR (no parallel TS adapter).

Slice plan after infrastructure lands

Rewrite of adapter-slice-plan under the Rust+WASM model. End state is the same (~180 sources, ~23 families); order shifts because every slice ships a Rust crate, not a TS file.

1. Slice 0 — infrastructure + jobs-ch port (above).
2. Slice 1 — Yousty Rust port (3 sources collapse into one Rust adapter crate; delete the TypeScript Yousty adapters when the Rust crate is registered).
3. Slice 2 — jobup-ch Rust port.
4. Slice 3 — Greenhouse multi-tenant family (Rust factory crate emitting N Adapter instances).
5. Slice 4 — SmartRecruiters + Lever bundle.
6. Slice 5 — Workday family.
7. Slice 6 — LinkedIn (needs runtime to inject a JS-capable Env.fetch — i.e. route through scrapers/router.ts based on adapter method).
8. Slices 7+ — Tier A long tail (Oracle HCM, SF HTML, SF SPA, Phenom, Personio, Ashby, Hibob, Workable, Recruitee, RSS family, Teamtailor, Refline, Prospective, Softgarden, Umantis).
9. Slices N+ — Tier B (Indeed, Xing, jobwatch, Stellenanzeiger, Adzuna, Reed) + Tier C (Swiss employers, niche boards).

Slices 1+2 are early because they remove the remaining TypeScript Swiss-board sources after equivalent Rust crates exist. The earlier the better.

What stays TypeScript forever

• lib/jobcache/src/contract.ts + adapter.ts — TS bindings to the WIT, hand-authored or generated. Stays for TS-side validators (ObservationSchema) and the host shim.
• lib/jobcache/src/ids.ts — TS-side adIdFromUrlLanguage, roleIdFromUrl, hashUrl, and hashText for scrapeUrl, browser, and the import path. Frozen, mirrored byte-for-byte in Rust.
• lib/jobcache/src/wasm-host.ts — TS host that loads .component.wasm and presents as Adapter.
• CareerVector scrapeUrl() / Jina / provider chain — one-shot, TS-native, stays.
• jobcache/ingest/src/scrapers/* (Router + Raw + Browserbase + Firecrawl) — device/runtime fetcher; the host wires a Router-backed Env.fetch into adapters that declare method: "headless-proxy". The cascade itself is TS.
• All UI surfaces (ui/, jobcache/interface/, jobcache/ops/ui/, jobcache/qa/ui/, jobcache/status/ui/) — Svelte.
• All other Node services unaffected.

Sketched Rust adapter — jobs-ch (proof shape, not final)

// crates/jobcache-adapter-jobs-ch/src/lib.rs

use async_trait::async_trait;
use jobcache::{
    Adapter, AdapterDescriptor, Env, Method, Observation, ObservationOptions,
    AdapterError, schema_org::parse_jobposting_observation,
    url_id::hash_text,
};
use regex::Regex;
use scraper::{Html, Selector};

pub struct JobsCh;

const ADAPTER_ID: &str = "jobs-ch";
const ADAPTER_VERSION: &str = "0.1.0";
const HOMEPAGE: &str = "https://www.jobs.ch";
const LISTING_URL: &str = "https://www.jobs.ch/en/vacancies/?page=0";
const MAX_DISCOVER_URLS: usize = 50;

#[async_trait]
impl Adapter for JobsCh {
    fn descriptor(&self) -> AdapterDescriptor {
        AdapterDescriptor {
            id: ADAPTER_ID.into(),
            version: ADAPTER_VERSION.into(),
            label: "jobs.ch".into(),
            homepage: HOMEPAGE.into(),
            method: Method::HtmlBespoke,
        }
    }

    async fn discover(&self, env: &dyn Env) -> Result<Vec<String>, AdapterError> {
        let body = fetch_text(env, LISTING_URL).await?;
        let urls = extract_detail_urls(&body);
        Ok(urls.into_iter().take(MAX_DISCOVER_URLS).collect())
    }

    async fn observe(
        &self,
        url: &str,
        env: &dyn Env,
        opts: &ObservationOptions,
    ) -> Result<Option<Observation>, AdapterError> {
        let started_at = now_ms();
        let body = fetch_text(env, url).await?;
        let final_url = body.final_url.clone();
        let html = &body.text;
        let content_hash = hash_text(html);
        let observation = parse_jobposting_observation(
            SchemaOrgInput {
                url,
                resolved_url: &final_url,
                html,
                source: Some(ADAPTER_ID),
                source_id: extract_source_id(&final_url).or_else(|| extract_source_id(url)),
            },
            ObservationOptions {
                adapter_id: opts.adapter_id.clone().or_else(|| Some(ADAPTER_ID.into())),
                adapter_version: opts.adapter_version.clone().or_else(|| Some(ADAPTER_VERSION.into())),
                content_hash: opts.content_hash.clone().or(Some(content_hash)),
                fetched_at: opts.fetched_at.clone().or_else(now_iso),
                observed_at: opts.observed_at.clone().or_else(|| opts.fetched_at.clone()).or_else(now_iso),
                ..opts.clone()
            },
            // The Rust schema_org helper also computes timing_ms / bytes_read from the input.
        );
        Ok(observation)
    }
}

fn extract_detail_urls(html: &str) -> Vec<String> { /* anchors + regex + __INIT__ parse */ unimplemented!() }
fn extract_source_id(url: &str) -> Option<String> { /* regex on /detail/<uuid-or-numeric>/ */ unimplemented!() }

(The full implementation lives in slice 0; this is the shape proof.)

Open decisions for Codex

1. WIT vs Rust trait as source of truth. I lean WIT for polyglot future-proofing; Codex picks based on tooling preference.
2. Async model. Component Model future<T> vs sync-fetch-bridged-from-host. I lean sync-fetch initially (simpler tooling, hosts already serialize), promote to async when wit-bindgen + jco support is stable.
3. WIT location. lib/jobcache/wit/ vs top-level wit/ vs crates/jobcache/wit/.
4. Rust workspace location. Top-level crates/ (peer to lib/, jobcache/, ui/, mcp/) vs jobcache/crates/. I lean top-level since the contract crate will be shared with CareerVector workspace work eventually.
5. WASM module storage. Committed under lib/jobcache/wasm/, built in CI and attached to release assets, or separately hosted. I lean CI-built modules plus explicit download/cache by the device/operator host to keep the repo lean.
6. CI runner. CircleCI per the existing convention. Cargo + wasm-tools + wasm-opt installed via toolchain action.
7. Native device/operator path. Today's trusted harness is Node+tsx loading WASM. Future option: a Rust device/operator binary linking the adapter crates directly. Out of scope for slice 0 but worth flagging — the Rust crate shape (crate-type = ["cdylib", "rlib"]) supports it without changes.
8. Cell payload value type. Rust serde_json::Value works but loses TypeScript's structural typing. Acceptable; the contract validates downstream.

What I'm waiting on before slice 1+

• WIT contract committed to a known location.
• jobcache Rust crate compiling with the contract types.
• wasm-host.ts shim loading a placeholder WASM.
• cargo build --target wasm32-wasip1 working in CI.

With those four, I can write the jobs-ch port + every subsequent family crate in parallel against a stable contract. Codex sign-off on this doc or counter-proposal kicks off slice 0.