JobCache ingestion and CareerVector consumer AI

These are two independent systems that share Sporewright's mathematics. They do not share a tensor, queue, identity, feedback log, or persistence boundary.

  • JobCache routes public corpus work and converges on one shared CrateDB.
  • CareerVector routes private product work for one workspace and learns in CareerVector D1.
  • In both systems the operator is a root-policy and resource owner. It is not an address segment and it is not a provider. Provider/model pairs and execution classes are the options that do work.
  • In both systems an attempt is evidence about a decision. It is not part of the address and it does not become a second scheduler beside the tensor.

1. JobCache: collection, fold, and enrichment

The solid paths below are present in the code. Dashed amber paths are the next extension: Cloudflare AI enrichment using the same task, receipt, deposit, and fold protocol rather than a separate data pipeline.

JobCache ingestion and Cloudflare AI enrichment The operator shapes a Sporewright orchestrator, which ranks work from CrateDB, publishes private RabbitMQ queues, and leases work to browser, desktop, and container participants. Participants fetch public portals and put transient observations into Corbet S3 with B2 overflow. The orchestrator folds those deposits through CrateDB COPY staging into current corpus tables. A proposed operator enrichment executor takes content-addressed embed tasks to Cloudflare Workers AI and returns results through the same deposit fold.

AS BUILT — resilient pull-based collection

Platform operator policy weights · budgets · catalogue · curiosity CrateDB corpus state ads · bindings · freshness trust · drift · lease ledger Sporewright server · alwaysdata rank due work · resolve addressed field mint lease, routing receipt, S3 grants queue.rs · tick.rs · router.rs · main.rs Private RabbitMQ priority queues per execution class no participant receives AMQP credentials Browser CORS-limited adapter dock Tauri desktop native fetch + render Containers lean + headless classes Public job portals and source APIs discover URLs · fetch · adapter parse · canonical facts Corbet S3 · transient deposits object-scoped PUT · same-key inbox plaintext public structured observation B2 overflow deposit + raw spool Raw archive RS03 encrypted Ledger-driven CrateDB fold POST /deposited or bounded recovery sweep COPY _deposit_staging → idempotent upserts clear exact lease key → delete transient object ingest.rs · cratedb.rs · object_store.rs One-node CrateDB · current corpus routing_outcomes · ad_observation_state · ad_field_state ad_facts · searchable ads projection · trust evidence raw archive stays in S3; transient deposit disappears only after fold bottom-up evidence Axiom sideways ops tap never read to route

NEXT — enrichment uses the same machinery Crate task ledger embed-text · input hash tasks + work_cache

Operator executor owns shared AI token pulls and batches leases Workers AI async Batch API pinned model epoch standard result deposit as built target extension

What the code actually does

Concern As-built implementation
queue inputs crates/sporewright-server/src/queue.rs derives due discovery and observation targets from CrateDB
routing crates/sporewright-server/src/tick.rs and router.rs resolve execution classes and publish bounded work
delivery rabbitmq.rs owns private per-class queues; main.rs /lease consumes and conditionally stamps the Crate ledger
participant write authority object_store.rs mints one expiring S3 PUT for one deposit key; no DB, AMQP, or bucket credential
durability Corbet S3 is canonical; raw_overflow.rs verifies encrypted-raw B2 promotion, while structured deposits use the same key in either inbox
corpus fold ingest.rs asks CrateDB itself to fetch each object with COPY _deposit_staging, then performs set-based upserts
current projections routing outcomes, current observation/field state, facts, searchable ad columns, and trust evidence
cleanup only the exact folded lease is cleared; only then is the transient deposit deleted
enrichment substrate jobcache/migrations/cratedb/schema.sql has embed-text, tasks, work_cache, text_chunks, embeddings, and job_vec; the current orchestrator queue does not yet activate that path

The new ingestion path is deliberately much thinner than the compatibility FactTreeStore command drain. Extraction participants already produce canonical facts before depositing. Cloudflare AI therefore belongs after the current observation has folded, where the content hash is stable and repeated scrapes cannot spend another inference on identical text.

D1, Durable Objects, Queues, or none?

None for the first version. The existing S3 mailbox and Crate task ledger already provide durable ingress and idempotency. The Workers AI asynchronous Batch API already queues non-interactive embedding work, returns a request ID, and accepts a payload up to 10 MB. The operator executor can persist that request ID in work_cache, poll it, and return a normal result deposit. That is the shortest path with the fewest truths.

Candidate Decision Reason
CrateDB tasks + work_cache use task state, content hash, model epoch, result pointer, and retries stay beside the corpus they describe
Corbet S3 deposit use already the authorized, replayable write boundary into the one corpus writer
Workers AI async Batch API use the provider already queues the batch; it avoids another Worker, database, and retry protocol
Cloudflare Queue reserve the correct Cloudflare primitive if we later need push delivery or provider-independent fan-out; it is still another at-least-once ledger today
D1 do not put in the ingest path a D1 database is another single-writer SQLite truth and the free database is capped at 500 MB; duplicating public ad bodies before CrateDB buys nothing
Durable Object do not use for bulk staging it is a single-threaded coordination actor, useful for a live shared token bucket or session—not a warehouse or ingestion queue

Cloudflare's current documentation supports these boundaries: Workers AI asynchronous Batch API, Queue batching and retries, D1 limits, and Durable Object coordination rules.

The vector-space gate

CrateDB currently declares 1024-dimensional corpus vectors while CareerVector's live query embedder emits 768-dimensional EmbeddingGemma vectors and explicitly skips kNN on a width mismatch. Corpus and query vectors must use one pinned model epoch; two embedding models do not share a meaningful vector space.

The least disruptive candidate is Cloudflare's 1024-dimensional @cf/qwen/qwen3-embedding-0.6b, because it fits the existing Crate columns and has a longer input window than the old BGE-M3 schema assumption. It must first beat the existing 400-ad retrieval benchmark. If it does not, migrate the Crate columns to the measured winner's width; never silently project, pad, or compare different spaces. See Cloudflare's current model dimensions.

2. CareerVector: consumer-facing Smart Routing

Here the provider APIs really are analogous to JobCache's workers. More exactly, our executors are workers: a local WebLLM runtime, a trusted workspace peer, a BYOK connector, or the platform operator's Included AI broker. Each executor offers provider/model options and returns receipt-bound evidence.

CareerVector consumer AI control plane Platform operator policy and workspace-owner policy add to a Sporewright addressed field. The field ranks local WebLLM, trusted workspace peers, BYOK provider connectors, and an operator Included AI broker. A receipt and short explanation accompany the decision. Outcomes return to a D1 projector and shape the field bottom-up without sending private content to CrateDB. Platform operator zero platform spend · shared quotas · allowlist quality floors · privacy gates · provider health owns Included AI keys; does not own workspace preference Workspace owner fixed / preferred / never · BYOK · consent quality appetite · curiosity · local resource budget explicit choices remain authoritative CareerVector D1 declared policy bounded evidence + budget no prompts, output, CV, or keys in events Sporewright addressed field workspace → capability → stage → consumer → instance? root policy + workspace residuals + learned belief + live gates options are provider/model execution routes; attempts are evidence Ranked decision + causal receipt selected route · fallbacks · hard gates user-facing: “Free credits exhausted; using Local AI” Local AI executor consented WebLLM model device-owned · credentialless · bounded tasks Workspace peer executor trusted P2P WebLLM capacity same workspace only for private work Your AI executor workspace BYOK connector user policy and budget; key stays scoped Included AI broker operator-hosted provider connectors shared free allowances; keys never leave server External AI providers provider/model options behind BYOK Free provider pool operator quotas, health, quality certification CareerVector pipeline extract · evaluate · tailor · explain Receipt-bound outcome projector failure · quality loss · latency · tokens dedupe, age, compact, message-pass upward Required key boundary before Included AI Public /chain returns route descriptors and availability only; executable shared keys remain server-side.

Who is what?

Thing Role in the model Not this
platform operator writes root policy, owns shared keys/quotas and quality/privacy floors not an address layer; not an AI provider
workspace owner writes authoritative workspace policy, consent, BYOK and curiosity budget not merely another learned signal
address workspace.capability.stage.consumer.instance?; selects the interacting branch not provider, model, key, attempt number, or runtime host
option a routable provider/model execution route not an individual browser or historical call
executor/worker local runtime, trusted peer, BYOK connector, or Included AI broker that realizes an option not the tensor itself
attempt one realized call, classified into normalized outcome dimensions not an alternative policy engine

The platform operator therefore belongs in the picture, but above the field and beside the server-owned executor—not inserted into every semantic address.

As-built gap that matters before shared Included AI

The adaptive route, receipts, attempt chain, learning projector, D1 budget authority, server /llm/* execution, and consent-first WebLLM runtime exist. Included AI and workspace-peer execution are not yet complete.

There is also one concrete security seam to correct first:

  1. resolveChainConfigForWorkspace() constructs an executable providerKeys map containing both workspace BYOK and entitled server keys in lib/server/src/chain-config.ts.
  2. GET /chain currently returns that entire payload to the browser. The route is in api/src/routers/misc.ts.
  3. The dashboard happens not to retain providerKeys, but transport already disclosed them; “the UI ignores it” is not a boundary.

Split the type into two views:

  • internal executable route: keys plus provider/model chain, readable only by the server executor;
  • public routing view: opaque route IDs, availability, consent/readiness, ranked explanation, and budget state—never shared credentials.

Browser-owned BYOK can remain browser-executable where the user explicitly chose that mode, but it must be loaded from a workspace-owned secret seam or local vault and merged client-side—not bundled with operator credentials in the public route descriptor. Operator-owned Included AI keys never cross the server boundary. Prompts, outputs, CV data, workspace identity, and LLM feedback never enter the JobCache CrateDB pipeline.

Consequence

The reusable principle is not “put every worker behind one central service.” It is:

one addressed field composes top-down policy with bottom-up evidence; every executor pulls or realizes work under a causal receipt; every result returns normalized feedback along that same path.

JobCache applies it to public data collection and enrichment. CareerVector applies it to private AI work. The operator participates in both as policy and resource authority, while the actual execution remains distributed.

Source: wiki/content/architecture/INGESTION-AND-CONSUMER-AI.md