Observation

JobCache receives @cv/jobcache Observation envelopes from top-down devices and safe bottom-up CareerVector imports. Both paths emit the same ad_id-centered shape; workspace jobs remain private.

There is one Cockroach write path: submitObservation. A JobCache device submits one Observation through a known active lease. A CareerVector workspace commit submits one Observation by enqueueing a D1 jobcache_commands submit_observation row in the same batch as the private job.create or job.update. JobCache drains that command and applies it through submitObservation. submitObservation validates schema and provenance; writes changed ad payload only when values move; persists emitted object_refs, chunk_refs, annotations, search_chunks, and embeddings as hot index rows; updates lightweight observation state on every accepted observation; updates global device trust on agreement or disagreement when a device is responsible; and queues verify-observation only on fresh conflict.

Role identity is URL-family identity: roleIdFromUrl(url). The German, French, English, and unknown-language ads for the same canonical URL are distinct ads under that role. Role payload is not adapter-owned. submitObservation writes current ad payload cells, records the URL-family role link, and performs only cheap ad_propagation for safe changed fields. It does not compare sibling ads, aggregate languages, or cluster roles by title/company. Cross-URL, cross-language, and semantic role reduction is later async database work. Read paths ignore stale direct or adapter-written role rows.

submitObservation does not fetch URLs, read object-store blobs, chunk text, enrich fields, create annotations, produce embeddings, or call LLMs. Those are device, import, or later task outputs. The server only performs cheap schema, provenance, policy, identity, hash, and reference validation before writing the records already present in the Observation envelope.

There is no extra intake, event, trust-scoring, or later fact-application layer. Do not replace the CareerVector D1 handoff with post-commit direct JobCache HTTP from /ops. If a direct trusted intake is added later, it is a separate ingestion surface with separate durability semantics.

The important boundary is payload versus metadata:

• payload is the actual job/ad/role content: why the system exists. The same sparse payload vocabulary is used at role, ad, and workspace-job level.
• metadata and state keep the system running: ids, URLs, language identity, lease/task ids, trust/churn/cooldown, timestamps, workflow status, files, and other node-specific machinery.
• A field that only makes sense on one tree level is metadata/state, not shared payload. status_id is workspace workflow state. Evaluations may later roll upward, but they are not part of this shared payload model until that reducer is explicitly designed.

Terms

Roles, ads, and workspace jobs form one sparse payload tree:

role.payload
  -> ad.payload
      -> workspaceJob.payload

Each level uses the same payload vocabulary and sparse cell shape. Node metadata differs by level. Roles and workspace jobs are not addressed by the submitted Observation.

Submit Path

flowchart LR
  observation[Observation submitted]
  submit[submitObservation]
  payload[changed ad payload]
  state[ad observation state]
  trust[device trust]
  verify[verification task]
  evidence[evidence / object refs / chunks / annotations]
  embeddings[Embedding records]
  reads[CareerVector / JobCache read surfaces]

  observation --> submit
  submit -- changed values --> payload --> reads
  submit -- every accepted observation --> state
  submit --> trust
  submit -- fresh conflict --> verify
  verify --> observation
  submit --> evidence
  evidence --> embeddings
  evidence -. operator review .-> payload

submitObservation is also where the lease is closed and task state is updated for device-origin Observations. For CareerVector-origin Observations, the drained D1 command is finished after the shared write succeeds.

For one submitted Observation:

1. Validate the Observation schema and payload vocabulary.
2. Validate either the active lease/task/device/session/adapter/resource policy or the trusted drained D1 command provenance for a CareerVector workspace write.
3. Reject workspace-private state and node metadata disguised as payload.
4. Derive canonical ad_id from normalized URL x language server-side, derive the active URL-family role_id from normalized URL server-side, then read each field key, cell, optional observed_at, and optional evidence_ref_ids.
5. Validate references across evidence_refs, object_refs, chunk_refs, annotations, search_chunks, and embeddings with cheap id/hash checks.
6. Upsert emitted support records into object_refs, evidence_objects, text_chunks, ad_annotations, and embeddings.
7. Compare the submitted value hash with the current ad payload metadata.
8. If there is no current value, write the submitted payload with provenance.
9. If the hash matches, leave the payload untouched, update lightweight ad observation state, and raise or preserve global device trust.
10. If the hash differs, write the submitted value as current payload. The last valid Observation wins the value.
11. If both sides are fresh and conflict, update device trust for disagreement and queue one verify-observation task so another device can overwrite again if needed.

Evidence explains the datapoint, but evidence does not provide trust by itself. Some payload fields are volatile. valid_through, description, workload, and salary can change during a listing's lifetime. URL identity, language identity, device/task ids, source-local ids, and workflow fields are metadata/state, not payload.

Device Trust

Device trust is global per device. Keep it simple:

device_id
success_count
conflict_count
verification_win_count
verification_loss_count
last_observed_at
last_conflict_at
trust_score

Agreement with current payload raises or preserves trust. Disagreement, policy violations, schema drift, hash mismatch, and failed verification lower trust. Trust is global per device in the current architecture. It is not field confidence.

Cheap State, Not Timeseries First

The RU-sensitive design separates hot payload from hot metadata/state and retains expensive evidence detail only when it has future value.

Hot Cockroach state should include:

• current ad payload for read-heavy content
• URL-family role links plus async reducer-owned role payload rows, never direct adapter role writes
• lightweight ad/ad-field state for last observation, last observer, churn, cooldown, and provenance hashes
• global device trust on the device record
• object, chunk, annotation, search-chunk, and embedding indexes by stable key

Cold object storage should include:

• retained raw HTML/API payloads
• screenshots where needed
• large extracted text
• replayable Observation payloads when worth keeping
• chunks whose text is too large or too redundant for hot rows

Boring confirmations should be cheap. If the incoming value hash matches the current payload and the evidence shape is ordinary, update trust and lightweight state without rewriting the content payload.

Conflict and change are expensive enough to justify retained evidence. They explain why the current value moved or why verification was requested.

RU Rules

• Compare hashes before reading or writing large payload.
• Keep identity and hot-index upserts no-op gated. Repeated role, ad, object, chunk, annotation, or embedding rows should not churn updated_at when the modeled values are unchanged.
• Return existing rows for duplicate enqueue/readback paths instead of forcing a write just to confirm existence.
• Expire stale leases in bounded oldest-first batches before selecting retryable work; do not run unbounded cleanup on the hot lease path.
• Batch device-trust updates where exact immediacy is not needed.
• Keep the changed-payload path one or two indexed reads plus one conditional write.
• Avoid hot SQL blobs for raw source material.
• Do not let devices query arbitrary JobCache state.
• Include a bounded current-value snapshot in the lease when comparison context is useful for the task.
• Make verification tasks specific: source, URL, ad_id, field, expected hashes, and evidence question.

The server stays dumb but shrewd. It does not recompute expensive work. It tracks identity, leases, hashes, current winners, device trust, and which outsourced verification would be worth more than its coordination cost.

Adapter Boundary

Adapters do not implement submitObservation, device trust, current-value policy, or role projection.

Adapters prepare source fragments containing:

• fields keyed by shared payload vocabulary
• cells with value hashes where available
• evidence refs
• object refs and chunk refs when produced
• annotations, search chunks, and embeddings when produced by the device/import
• stale or closed markers represented in shared vocabulary
• source-local diagnostics and structured errors

If an adapter sees contradictory source material, it should represent the observed fields and evidence. The broker decides whether that becomes a current value change, a verification task, or a device-trust update.