Cloudflare AI Gateway fit for CareerVector

Status: research memo, 2026-05-19. This note evaluates Cloudflare AI Gateway against CareerVector's AI model: browser/client BYOK calls where possible, server-key calls only for managed, server, fallback, QA, admin, and shared ingestion paths.

Recommendation

Adopt AI Gateway only for CareerVector-controlled calls first. It is a good fit for server keys, server fallback, QA/admin probes, Sysadmin experiments, and JobCache enrichment where CareerVector owns the bill and needs operator visibility. It is not a clean fit for pure browser BYOK calls because authenticated gateway use requires a Cloudflare run token, while unauthenticated gateway use invites abuse and log noise.

Do not route normal user-owned browser BYOK traffic through AI Gateway unless CareerVector intentionally introduces a server proxy for that feature. That proxy would make CareerVector a handler of user provider keys and prompt/response traffic, which is a different privacy and support contract.

What AI Gateway can do

Official Cloudflare docs describe AI Gateway as an AI proxy for observability and control across model providers. Current useful pieces:

Capability	Cloudflare claim	CareerVector value
Provider routing	REST API can call Cloudflare-hosted and third-party models through Cloudflare API endpoints, with OpenAI-compatible chat and responses endpoints.	One server-side call shape for OpenAI/Anthropic/Google/Workers AI experiments.
Analytics	Dashboard and GraphQL expose requests, tokens, cache, errors, and cost metrics.	Feed Ops FinOps and Analytics with a second source beside D1 `ai_usage` and `gateway_log`.
Logging	Logs can include prompt, response, provider, timestamp, status, tokens, cost, and duration. Per-request headers can disable the full log or store metadata only.	Metadata-only logs for green/yellow tier; payload logs only for red/debug sessions.
Custom metadata	Up to five string/number/boolean metadata entries per request can be logged and filtered.	Attach route, workspace hash, actor class, function, and environment without leaking full workspace content.
Caching	Exact-match response caching for text and image responses; default cache key includes provider, endpoint, model, provider auth header, and full request body.	Useful for deterministic QA prompts, model health probes, repeated classification fixtures, and shared JobCache prompts. Weak for personalized CVL/job prompts.
Rate limiting	Gateway-level fixed or sliding request limits return `429` when exceeded.	Budget guard for server keys and admin experiments.
Request handling	Per-request timeout and retry headers; retries max at five attempts.	Edge-level resilience for managed calls, but CareerVector's existing cascade should stay the primary behavioral contract until deliberately replaced.
Dynamic routing	Beta route flows can branch on body, headers, or metadata, apply rate/budget limits, and choose model fallbacks.	Possible future control plane for server-side model chains after parity with current cascade is proven.
BYOK store	Cloudflare can store provider keys in Secrets Store and reference them from gateway configuration.	Good for CareerVector-owned provider keys. Not the same thing as letting every workspace user bring a private browser key.
DLP and guardrails	DLP can scan prompts/responses; Guardrails can flag or block harmful content. DLP response scanning buffers streaming responses.	Use selectively for admin/red-tier or server-managed calls; do not put it on latency-sensitive browser streaming by default.

Browser BYOK fit

Pure browser BYOK is a poor first target.

CareerVector's normal browser model keeps user provider keys and calls as close to the user's session as possible. AI Gateway changes that boundary:

Authenticated gateways require a valid Cloudflare API token per request. Cloudflare's docs recommend authenticated gateways to prevent unauthorized access and invalid requests that inflate log storage. Exposing that Cloudflare token in the browser is not acceptable.
Provider-native gateway calls can still pass a provider API key in the request, but an authenticated gateway also needs cf-aig-authorization. Without authentication, the gateway URL becomes an abuse surface.
Cloudflare BYOK stores provider keys in the Cloudflare account. That is useful for CareerVector-managed keys, not for arbitrary per-workspace user keys that should not become operator-managed account secrets.
AI Gateway logs are enabled by default. Even with cf-aig-collect-log-payload: false, Cloudflare still logs metadata such as tokens, model, provider, status, cost, and duration. That is fine for managed traffic, but it should be a deliberate consent-tier decision for user-owned browser traffic.

The clean browser-BYOK answer is therefore: keep direct browser-to-provider calls where CORS/provider policy allows it, keep CareerVector's cascade and local telemetry for user-visible execution, and use server fallback only when the browser cannot complete the work.

CareerVector call paths that fit

Server keys

Best first fit. CareerVector already distinguishes server-key readiness in Ops FinOps. Calls made with operator-funded env/D1 keys can safely route through an authenticated AI Gateway or the Cloudflare REST API. Use stored provider keys or unified billing only for keys CareerVector owns.

Server fallback

Good fit, with care. MCP/headless dispatch and stuck-request recovery are server-bounded and already visible in Workspace Intelligence as server-fallback. AI Gateway can add retry, rate-limit, cost, and error visibility. The existing callWithChain cascade should remain authoritative until a route-by-route parity test proves Dynamic Routing can express the same provider order, cooldown, critic rejection, abort, attribution, and fallback semantics.

QA

Strong fit. QA probes, deterministic fixtures, and model-health checks are operator-funded, repeatable, and naturally inspectable. Exact-match caching can reduce repeated fixture cost. Payload logging can be enabled for non-user fixtures, while production workspace QA should use metadata-only logs unless red-tier debug is active.

Sysadmin

Strong fit for experiments and operator workflows. The Sysadmin tab can use AI Gateway for controlled model tests, provider comparisons, DLP/guardrail experiments, and prompt/response inspection because the operator is the principal and the calls are not normal user BYOK.

JobCache

Good fit for shared ingestion and enrichment. JobCache is the top-down shared role/ad corpus; CareerVector pays that cost. AI Gateway metadata can tag source, scraper, role/ad id hash, task, and environment. Caching is more plausible here than for CVL because shared ads and extraction prompts repeat more often. Do not cache prompts that include workspace-private notes or CVL content.

Inspector and Events

Fit as read surfaces, not write authority. The cockpit should ingest AI Gateway analytics/log summaries into normalized Ops API shapes. It should not make the UI speak Cloudflare-specific APIs directly, and it should not replace existing D1 ledgers. AI Gateway is a reconciliation and drill-down source, not the canonical workspace execution log.

Metadata and logging strategy

Use separate gateways by traffic class:

Gateway	Traffic	Default logging	Payloads	Notes
`cv-managed-prod`	server fallback, managed server keys	on	off by default	Primary FinOps source for controlled production calls.
`cv-qa`	QA fixtures and synthetic probes	on	on for non-user fixtures	Safe place to test caching, retries, DLP, and guardrails.
`cv-debug`	red-tier support sessions	on	on only while consent window is active	Short retention and explicit operator workflow.
`jobcache-prod`	shared ingestion/enrichment	on	off unless source content is public and already in JobCache evidence	Keep JobCache attribution separate from workspace-private calls.

Use cf-aig-metadata with at most five entries:

surface: server_fallback, qa, sysadmin, jobcache, mcp
function: extract, enrich, evaluate, tailor, health_probe
actor: server, agent, operator, jobcache
workspace: hashed/truncated workspace id, or none
env: prod, dev, qa

For green/yellow telemetry tiers, send cf-aig-collect-log-payload: false so AI Gateway stores metadata without raw prompts/responses. For fully private or high-risk calls, use cf-aig-collect-log: false and rely only on CareerVector's own sanitized D1 ledgers. For red-tier support, payload logging may be enabled for a bounded consent window and then disabled again.

Ops should continue to read:

D1 ai_usage as the internal token/call ledger.
D1 gateway_log for CareerVector's service-provider attempts.
AI Gateway analytics/logs as an external reconciliation source.
Provider billing APIs for final cost truth where admin keys exist.

Risks, security, and cost

Browser token exposure: authenticated AI Gateway needs a Cloudflare token. Do not ship that to browsers.
Untrusted gateway abuse: unauthenticated gateways are not suitable for public browser traffic. Cloudflare explicitly recommends authentication when storing logs.
Privacy drift: logs are enabled by default. If payload logging is not controlled per gateway or per request, prompts and responses can become visible in Cloudflare logs.
Consent mismatch: metadata-only logs may still expose provider, model, token count, status, cost, and timing. Treat this as operator telemetry and align it with the green/yellow/red model.
Free-plan log cap: Workers Free has 100,000 logs total across all gateways; Workers Paid has 10,000,000 per gateway. Use auto-delete or explicit retention, and do not treat AI Gateway as the only ledger.
Caching correctness: AI Gateway caching is exact-match and cache is volatile. Custom cache keys can accidentally cross user/workspace boundaries if chosen badly. Keep cache disabled for workspace-private calls unless the cache key includes a privacy-safe scope and a stable prompt version.
DLP latency: DLP response scanning buffers streaming responses, which hurts time-to-first-token. Use separate gateways or request-only DLP for latency-sensitive traffic.
Dynamic routing maturity: Dynamic Routing is powerful but beta in the current docs. Do not move CareerVector's cascade policy there in one step.
Cost estimation: AI Gateway cost metrics depend on providers returning token data and model names. Provider dashboards and billing APIs remain the authoritative spend source.

Staged adoption plan

Stage 0: no product behavior change

Create no runtime dependency yet.
Add Ops API fields for optional AI Gateway readiness: gateway configured, log mode, and last analytics sync.
Decide gateway names and metadata keys.
Keep browser BYOK untouched.

Initial implementation note: /finops/summary now exposes managedAiGateways[] as the vendor-neutral readiness surface for this. The current env contract is:

CV_AI_GATEWAY_ACCOUNT_ID (falls back to CV_CF_ACCOUNT_ID for readiness)
CV_AI_GATEWAY_ID
CV_AI_GATEWAY_API_TOKEN (falls back to CV_CF_API_TOKEN for readiness)
CV_AI_GATEWAY_LOG_MODE (metadata_only, payload, or disabled)

Stage 1: QA and Sysadmin gateway

Route synthetic QA/model-health calls through cv-qa.
Add a Sysadmin-only "gateway probe" action that calls a cheap model, verifies metadata appears, and checks payload logging policy.
Test cf-aig-collect-log-payload: false, request timeout, retry headers, and exact-match caching on non-user fixtures.

Stage 2: managed server-key calls

Route one low-risk server-managed function through cv-managed-prod. Good candidates: admin provider health, server-only extraction fallback, or another path that already uses CareerVector-owned keys.
Write both internal D1 usage rows and AI Gateway metadata so FinOps can compare them.
Keep the existing cascade in code; AI Gateway is the transport and observability layer only.

Stage 3: server fallback and MCP/headless

Route MCP/headless fallback calls through AI Gateway where the key is CareerVector-owned.
Preserve existing attribution: requested-by class, claimed-by class, provider/model, attempts, selected provider, and wasFallback.
Surface AI Gateway log id or event id in the process-request result only when the workspace telemetry tier permits it.

Stage 4: JobCache shared ingestion

Add jobcache-prod for shared ad/role enrichment and QA around scrapers.
Use metadata keyed to source/task/public ad id hash, not workspace id.
Enable caching only for stable public-source prompts and versioned extraction/classification fixtures.

Stage 5: evaluate Dynamic Routing

Mirror one existing provider chain into a Dynamic Route in shadow mode.
Compare provider order, fallback reasons, latency, cost, and output quality against callWithChain.
Promote only if the route can be rolled back instantly and the Ops cockpit still sees the same normalized attempt data.