Operator policy
Weights, budgets, source catalogue, and curiosity shape the routing field.
JobCache operatorBlueprint · domain
Public corpus ingestion
Resilient pull-based work distribution over unreliable actors, with raw evidence preserved before canonical publication.
Cell 3.1 · runtime · data · control · evidence- Operator policy to Sporewright router: shape
- Sporewright router to Private RabbitMQ: rank + lease
- Private RabbitMQ to Execution pool: pull
- Public job sources to Execution pool: fetch + parse
- Execution pool to Corbet S3 mailbox: deposit
- Corbet S3 mailbox to Content-addressed enrichment: pull immutable input
- Content-addressed enrichment to Canonical COPY fold: validated result
Sporewright router
Ranks due work and actor class; mints lease and causal receipt.
Open layer ↘Private RabbitMQ
Priority lanes carry work, never participant credentials.
JobCache orchestratorPublic job sources
Portals, feeds, APIs, discovery, and refresh signals.
Public webExecution pool
Browser, Tauri, and container actors parse locally and return feedback.
JobCache participantsCorbet S3 mailbox
Scoped transient deposits, immutable raw archive, and B2 overflow.
JobCache ingestionContent-addressed enrichment
Fingerprint, extract, embed, validate, and reconcile once per input and recipe.
JobCache pipelineCanonical COPY fold
Receipt validation and idempotent batch publication into Core.
JobCache Core01 · policy · mixed Operator policy Weights, budgets, source catalogue, and curiosity shape the routing field.
JobCache operator- 02 · process · as-built Sporewright router
Ranks due work and actor class; mints lease and causal receipt.
Open deeper layer ↘ 03 · service · as-built Private RabbitMQ Priority lanes carry work, never participant credentials.
JobCache orchestrator04 · actor · mixed Public job sources Portals, feeds, APIs, discovery, and refresh signals.
Public web05 · process · mixed Execution pool Browser, Tauri, and container actors parse locally and return feedback.
JobCache participants06 · store · as-built Corbet S3 mailbox Scoped transient deposits, immutable raw archive, and B2 overflow.
JobCache ingestion07 · process · target Content-addressed enrichment Fingerprint, extract, embed, validate, and reconcile once per input and recipe.
JobCache pipeline08 · boundary · as-built Canonical COPY fold Receipt validation and idempotent batch publication into Core.
JobCache Core
Flows
- Operator policy → Sporewright router shape
- Sporewright router → Private RabbitMQ rank + lease
- Private RabbitMQ → Execution pool pull
- Public job sources → Execution pool fetch + parse
- Execution pool → Corbet S3 mailbox deposit
- Corbet S3 mailbox → Content-addressed enrichment pull immutable input
- Content-addressed enrichment → Canonical COPY fold validated result
Ingestion
Outcome
Continuously acquire public job observations, preserve what was seen, extract and enrich it once per stable input, and publish a coherent result into JobCache Core without overwhelming the production database.
Pipeline
discover → lease → fetch → preserve raw → fingerprint → extract → enrich → validate
→ reconcile ad/role → publish batch → verify → age and refresh
Authorities
| Concern | Authority |
|---|---|
| Raw observations | Corbet S3 with declared overflow/fallback storage |
| Unfinished work | Small rebuildable pipeline state, not CrateDB truth |
| Routing and feedback | JobCache's Sporewright application |
| Embeddings and selected analysis | Cloudflare AI behind content-addressed caches |
| Published public facts | CrateDB in JobCache Core |
Invariants
- Preserve raw evidence before expensive or lossy transformation.
- Never perform the same expensive transformation twice for the same stable input and recipe.
- Actors pull bounded leases; the orchestrator does not assume reachability or capability.
- Browser, desktop, and container actors return the same receipt and feedback contract.
- Failed enrichment cannot partially mutate the canonical role/ad projection.
- Interactive workloads outrank asynchronous replenishment.
Current state
The adaptive extraction orchestrator, browser/Tauri/container participants, routing receipts, S3 preservation, CrateDB deposit path, evidence aging, and source adapters exist in varying degrees of maturity. The complete cold processing ledger and atomic publish pipeline remain target evolution.
Completion evidence
Load, drift, poison, retry, duplicate, outage, and heterogeneous-actor simulations plus a production trace from source discovery to a published role revision.