Broker Flow

HISTORICAL (broker lane abandoned). The active transport is the orchestrator: devices POST /lease for targets (including discover targets), scrape, deposit results to B2, and notify via POST /deposited; discovered child URLs are submitted via POST /discovered; ingest is ledger-driven (no B2 LIST). The Render planner/queue lane documented below is abandoned — see the root wiki SCRAPER-FLEET §6 ("There is no Render queue. There never will be again.") and CONTROL-PLANE. The broker code remains in the tree as fallback only; this page is kept as the record of that path.

This was the operational path for distributed JobCache ingest. It starts with a source catalog entry and ends with current ad payload plus lightweight state attached to the shared ad_id/field/cell boundary.

Adapters do not own this path. Claude owns source/family adapter code. Codex owns the contracts, broker, DB schema, submitObservation, QA, ops, and status surfaces that make adapter output safe to use.

End-To-End Path

sequenceDiagram
  participant Planner
  participant Cockroach
  participant Broker as Interface broker API
  participant Device
  participant Adapter
  participant Submit as submitObservation
  participant ObjectStore as S3-compatible cold evidence

  Planner->>Cockroach: source + binding due -> task row
  Device->>Broker: POST /api/devices/capability
  Broker->>Cockroach: upsert device + leader session
  Device->>Broker: POST /api/devices/lease
  Broker->>Cockroach: lease pending task
  Broker-->>Device: lease with bounded task
  Device->>Adapter: execute adapter for task
  Adapter-->>Device: fields + emitted refs/chunks/annotations/embeddings
  opt retained raw evidence
    Device->>ObjectStore: write bounded blob with scoped capability
    Device-->>Adapter: object ref metadata
  end
  Device->>Broker: POST /api/devices/observations
  Broker->>Submit: validate schema, lease, and policy
  Submit->>Cockroach: write changed ad payload and emitted hot index rows
  Submit->>Cockroach: update global device trust
  opt fresh field conflict
    Submit->>Cockroach: queue verify-observation task
  end
  Submit->>Cockroach: close lease and update task state

The first implemented broker surface lives on the interface service:

• POST /api/devices/capability
• POST /api/devices/lease
• POST /api/devices/observations

These routes are bearer-token gated and server-side only. A device never receives Cockroach credentials. A device should not receive permanent object-store credentials either; when cold evidence upload is needed, it should receive a bounded upload capability or upload through a server-controlled path.

Records In Play

The control-plane rows are:

The JobCache support rows are:

The business record remains the shared tree and current field/cell state in Cockroach. Cold evidence is replaceable. Losing object storage hurts replay, operator review, and re-extraction; it must not break day-to-day CareerVector workspace operation.

Task Creation

The planner reads sources, adapters, and bindings, then creates small tasks.

Task creation should obey these rules:

• one task is bounded by URL/source input, byte limit, timeout, task class, and expected Observation schema
• input_hash is stable so duplicate work can be suppressed
• duplicate enqueue returns the existing task row when no mutable task fields changed; it must not bump updated_at just to prove the task exists
• known_content_hashes lets devices skip unchanged content
• source_hints may include public selectors, API paths, sitemap patterns, or family-specific hints
• the planner decides coverage and freshness; devices do not choose the crawl frontier
• expensive operations can become separate tasks, for example chunk-text, enrich-observation, embed-text, or verify-observation

This keeps source coverage policy out of adapters and device code.

Lease Rules

The broker leases pending work to the current device leader.

Minimum lease rules:

• only a device/session that has submitted a fresh capability report can lease work
• task class must be one of scrape, chunk, enrich, embed, or verify and must match the device's allowed_task_classes
• resource policy must fit the device's current state
• leases expire and can be retried by another device
• one task has at most one active lease
• devices report capacity; the broker decides assignment
• before selecting ready work, the broker may expire a small oldest-first batch of overdue active leases, returning retryable tasks to pending and marking exhausted tasks failed

The default should be conservative: one active task per task class per device unless the resource policy explicitly allows more. Capability reports separate supported_task_classes from allowed_task_classes: a laptop can support enrich or embed in principle while omitting those classes from allowed_task_classes on battery, metered network, thermal pressure, or high load.

Observation Submission

A device submits an observation. It does not submit SQL.

For scrape tasks, the observation should include:

• stable task_id, lease_id, device_id, and session_id
• adapter_id and adapter_version
• url, resolved_url, url_hash, content_hash, and observation_hash
• timing, byte count, and resource metadata
• ad_id
• fields, an object keyed by shared field vocabulary; each field has a cell and optional evidence refs/observed_at
• optional object_refs for retained cold evidence
• optional chunk_refs for raw/stable chunks
• optional annotations and search_chunks
• optional embeddings for chunks processed during the same run
• structured errors when partial or failed

Embedding-only tasks still submit an Observation anchored to the ad_id, with empty fields and populated embeddings. Each Embedding record identifies the chunk, model pack, tokenizer, dimensions, vector hash, and vector payload. Embeddings are not renamed into observations; they ride inside one when a device task produced them.

Adapters emit the shared Observation shape directly. That submitted shape is shared by top-down JobCache and safe bottom-up CareerVector public-ad paths. For CareerVector-origin workspace writes, the Observation is enqueued as a D1 jobcache_commands submit_observation row inside the same workspace job.create/job.update commit batch. JobCache drains the command and applies it through the shared writer. /ops must not send those Observations with post-commit direct JobCache HTTP. A future direct trusted intake would be a separate ingestion surface, not the workspace commit handoff.

submitObservation

submitObservation is the only Cockroach write path for submitted Observation output. It is cheap server-side validation plus current value application and hot-index persistence. It is not server-side fetching, chunking, embedding, enrichment, LLM extraction, object-store readback, or graph derivation. Device Observations arrive through an active lease. CareerVector-origin Observations arrive through trusted drained D1 commands that were committed with the workspace op.

submitObservation validates:

• the shared Observation schema
• either the active lease, task, device, and session, or trusted drained command provenance for CareerVector-origin workspace writes
• adapter/task identity and resource policy
• evidence refs, object refs, chunk refs, annotations, search chunks, embeddings, and field vocabulary
• absence of workspace-private data

Then it writes the current shared state directly:

• write changed ad payload for non-conflicting fields
• persist emitted object_refs, chunk_refs, annotations, search_chunks, and embeddings into object_refs, text_chunks, ad_annotations, and embeddings hot index rows
• use no-op-gated identity and index upserts: if the derived role, ad, object, chunk, annotation, or embedding row is byte-for-byte equivalent at the modeled fields, Cockroach should not rewrite the row
• update global device trust on agreement or disagreement
• close the lease and update task state for device-origin Observations, or finish the drained command for CareerVector-origin Observations
• queue a verify-observation task only when a fresh conflict needs another independent device

Invalid schema, stale lease, or policy failures fail the submission path and leave current ad payload unchanged.

See observation.md for the conflict and trust rules. The short version is: the last valid Observation wins the current payload cell, fresh conflicts queue bounded verification, and device-trust decisions stay out of adapters.

This is where top-down and bottom-up meet. A JobCache device Observation and a CareerVector URL import should produce the same ad_id/field/cell shape after private workspace fields are stripped. submitObservation decides whether those fields reinforce current values, replace stale values, or require a verify-observation task.

Ops And QA

Ops needs to answer these questions without reading adapter code:

• which sources are due, paused, failing, or expensive
• which bindings are producing stale or low-trust observations
• which devices are active, trusted, limited, or suspended
• which tasks are pending, leased, expired, verifying, or failed
• which submissions changed current values or queued verification
• which object refs are missing or stale
• which public ad cells or role projections changed

QA should cover the broker path with fake devices before broad source rollout:

• capability report validation
• one active device leader
• lease creation and duplicate-lease prevention
• expired lease retry
• schema, lease, policy, duplicate, and verification-request outcomes
• observation persistence without direct DB access from the device
• direct current value writes from submitted fields
• no durable local filesystem evidence references

Ownership

Claude owns:

• Adapters
• family helpers
• source fixtures
• source-specific parser behavior
• adapter versioning
• adapter conformance tests

Codex owns:

• shared JobCache and device contracts
• broker API shape
• device control-plane DB schema
• submitObservation current value writes, conflict handling, and global trust
• ops, QA, and status surfaces
• integration review

If an adapter needs a contract change, the adapter owner should stop and write the gap down. Contract expansion belongs in the Codex-owned architecture track.

Reference Guideline

Use the source-reference guideline consistently:

• dark grey: study source coverage, workflow shape, reliability tactics, stale checks, dedup behavior, and operational loops closely enough that JobCache can compete
• black: import another product's code, prose, fixtures, arbitrary constants, source registry, or flat vacancy data model as canonical JobCache material