Your functions call models. Something should route, meter, and cap those calls —
without becoming another service to deploy. The riz gateway is
already in the binary serving your functions:
point any OpenAI client at /_riz/v1 and you're governed.
One wire format — the OpenAI shape every SDK, LangChain, LlamaIndex, and CrewAI
already speak. Behind it, riz routes by model prefix
(anthropic/claude-… → Anthropic, mapped to the
Messages API for you) with a de-duplicated fallback chain when a provider is down.
mock for CItools/tool_choice → tool_calls, on every provider — Anthropic tool_use/tool_result mapped for youbase_url — that's the whole migrationriz init ai-chat scaffolds one end-to-end — see it on Examples[agent] block runs this loop as a delegable A2A agent — the agent layerchat_completions_returns_openai_shape ·
models_lists_configured_providers ·
chat_completions_with_tools_returns_tool_calls ·
chat_completions_tool_result_turn_returns_text ·
streaming_returns_openai_sse_chunks
[gateway] default_provider = "anthropic" fallback_chain = ["anthropic", "openai", "ollama"] budget_usd = 50.0 # hard stop · HTTP 412 on breach [gateway.providers.anthropic] kind = "anthropic" api_key_env = "ANTHROPIC_API_KEY" [gateway.providers.openai] kind = "openai" api_key_env = "OPENAI_API_KEY" [gateway.providers.ollama] kind = "ollama" # local · no key · free
budget_usd is a hard stop: breach it and the gateway
answers HTTP 412 instead of spending. And the detail
that separates a real cap from a decorative one: a model the pricing table doesn't
know is billed at the most expensive tier — an unlisted model can never silently
bypass your cap. Local Ollama models are known-free.
GET /_riz/v1/usage is the ledger: cumulative cost and
tokens per provider. The same numbers flow into the OTLP pipeline as OTel GenAI
token attributes, rolled up per request — so "what did this agent chain cost?"
has an answer.
budget_zero_rejects_with_412 ·
unknown_model_fails_closed_at_the_most_expensive_tier ·
multi_hop_agent_chain_rolls_up_token_usage_across_the_tree
{
"anthropic": {
"requests": 412,
"tokens_in": 88210,
"tokens_out": 31455,
"cost_usd": 3.62
},
"ollama": { … "cost_usd": 0.0 }
}
# over budget?
POST /_riz/v1/chat/completions → 412
Every /_riz/v1/* route is bearer-gated with a
constant-time compare when a token is configured — the same gate as the rest of the
admin surface. A leaked URL without the token gets a 401, not your invoice.
stream: true is
true token-level streaming on every real provider: OpenAI-compatible upstreams
(OpenAI, Ollama, any base_url) pass through byte-for-byte
(streaming_passes_upstream_chunks_through_verbatim),
and Anthropic's SSE events are translated on the fly into OpenAI chunks — text deltas,
streaming tool_calls, exact usage
(anthropic_stream_translates_to_openai_chunks).
Both are pinned in the
claims registry.
The deterministic mock still answers buffered and is re-emitted as standard SSE chunks —
it has no tokens to stream.
gateway_endpoints_return_401_without_token_when_configured
# the whole migration: client = OpenAI( base_url="http://localhost:3000/_riz/v1", api_key=os.environ["RIZ_AUTH_BEARER_TOKEN"] ) r = client.chat.completions.create( model="anthropic/claude-opus-4-8", messages=[{"role":"user","content":"hi"}] )