Gateway and routing

Where things are

Old links under /settings/routing redirect to Gateway › Router, /settings/routing/litellm to Gateway › Connect, /settings/routing/prompt-experiments to Experiments and /routing/runtime to Gateway › Runtime.

Set up routing

1. Store an inference key

Open Gateway › Connect › Inference credentials and choose Add credential. Pick OpenAI, Anthropic or Google Gemini, paste the Inference API key and choose Store key.

• Keys are encrypted and cannot be read back. Each provider has one key; Rotate key replaces it.
• Billing-admin credentials used for billing imports are never reused for inference.
• For Google Gemini, use a Gemini API key from Google AI Studio. Vertex AI service accounts are not supported.
• The router's own internal calls (the AI task classifier, guardrail and semantic-cache embeddings, evaluation candidates and judges) also use these keys. Each one is metered and shows as its own decision.

2. Approve each model

On the Model capabilities tab choose Add capability, pick the Provider model from the priced catalog and enter Max context tokens, Max output tokens and whether it supports Tools, Structured output and Vision. Choose Save capability. Saving the same model again overwrites its record.

The router refuses any model without a record (model_capability_not_configured), and checks every request against it: tools, response_format and images need the matching flag, the output cap must fit Max output tokens, and the prompt plus output must fit Max context tokens. Enter only what you have verified for your provider account.

3. Send traffic

Point your app at the router with an ingest key (create one under Governance › Keys); see Router. With no policy, claude- models go to Anthropic and every other model to OpenAI, exactly as requested.

4. Check what happened

Each request becomes a decision on Improve › Routing and in Latest decisions on the Gateway overview:

The usage itself appears with your other usage, with exact tokens and cost and the attribution fields the request carried.

Routing policies

A policy sends matching requests to a model you choose. Create one with Add routing policy on Gateway › Router:

Creating or resuming a policy checks that its target and fallback providers have an inference key and a capability record, that a cost ceiling has prices to estimate against, and that an Automatic policy has tier assignments to use.

How the router picks a policy. It takes the enabled policies whose requested model, action and feature match, orders them by priority and then newest first, and uses the first whose rollout includes the request. Rollout buckets are stable per policy version, API key and rollout key (the X-TokenMeter-Rollout-Key header, else customer_ref, user or trace_id), so one conversation stays on one model. When nothing matches, the requested model is used as sent.

Matches every request. A policy with no requested model, action or feature applies to every request, including requests for other providers' models, and is marked with this badge. A policy that can never win because a higher one always matches first is marked Never reached: vN always wins first. Requests to /v1/messages that a policy would send to a non-Anthropic model are refused, and the policy says so.

Which policy wins? opens a tester: enter a requested model, the endpoint (OpenAI-compatible or native Anthropic), action, feature, API key and rollout key, and choose Check. It shows the model that serves the request, the matching policies in the order the router tries them, and any live experiment or evaluation canary that applies. It uses the same code as the router. Fallbacks, open circuits and monthly-budget downgrades can still change the model at request time.

Changing a policy. Policies are versioned and not edited in place.

• Ramp raises the rollout on the same version. People already in the rollout stay in; lowering is not offered.
• Retire stops the policy.
• Resume turns a paused policy back on after the same checks as creation.

Automatic pause. A new policy pauses itself when, in one hour with at least 50 decisions, its failure rate goes above 5%, its fallback rate above 10%, or its p95 latency above the optional guardrail (or success rate or quality score below theirs, once enough outcomes are linked). It then shows paused by guardrail, and the trip is listed under Gateway › Runtime › Guardrail trips. Check the decisions before choosing Resume.

Cost ceilings. Before calling the provider the router works out the request's worst-case cost: the exact input token count plus the output cap (the request's max_tokens or max_completion_tokens, else the model's Max output tokens). If that is above the tighter of the policy's Max request cost USD and the workspace's Per request, at most (Settings › General › Defaults), the request is refused with 402 and both amounts. Callers do not need to send anything for this.

Fallbacks, circuit breaker and timeouts

Fallback. When the target model fails at the network level, times out, returns 408, 429, 500, 502, 503 or 504, or its circuit is open, the router tries the policy's fallback model once. The decision's reason becomes policy_fallback. The fallback is skipped when:

• the request has a run_id;
• a governance token or cost cap applies to it, or a governance policy blocks the fallback model;
• the fallback's worst-case cost is higher than the amount the request was already checked against;
• it is a /v1/messages request and the fallback is not an Anthropic model;
• the response has already started streaming (the choice is made before the first byte is sent).

Other provider errors are returned as they are. The router never retries the same model.

Circuit breaker. Each workspace has a circuit per provider and model. Three of those transient failures in a row, each within two minutes of the last, open it for 60 seconds. While it is open, requests go to the fallback if there is one, and otherwise fail with 503 provider_circuit_open and Retry-After: 30. A successful call closes it. Gateway › Runtime › Provider circuits lists routes with recent failures, and the Gateway overview shows Degraded while any circuit is open.

Timeouts. A non-streaming call has 30 seconds. A streaming call has 30 seconds to start, 60 seconds between chunks and 10 minutes in total. These are not configurable from the dashboard. If a stream breaks part-way, the response ends, the decision is marked failed and the tokens it did produce are still metered.

Tiered routing

Tiered routing sends each request to a model chosen for its kind of task. Task types are grouped in three tiers, Light, Mid and Premium, shown as columns under Gateway › Router › Tiered routing with each task's assigned model.

1. Choose a Goal and Regenerate assignments. Cost picks the cheapest model that clears a minimal quality bar, Balanced the cheapest that clears a solid bar, Quality the highest-capability model with cost as tiebreaker. It runs as a background job.
2. A model is assigned only with independently judged shadow evidence: at least 200 paired responses, acceptable quality and critical-failure rates, format pass rate, measured cost savings and latency. Tasks without enough passing evidence show Not yet generated. Open a task to see its measured cost and quality and the candidates considered.
3. Pin override fixes a task to a provider and exact model (it needs a capability record) and protects it from regeneration. Unpin releases it.
4. Create a policy with Tiering Automatic. Nothing changes until you do.

Each request on an Automatic policy is classified first. A task type with no assignment makes the request fail with 400 tiered_routing_not_configured, rather than guessing a model.

Request handling

The Request handling card on Gateway › Router holds four workspace settings, saved with Save changes:

• Prompt-injection guardrail. Scans every user message and tool result before routing. Clear-cut injections are blocked by pattern rules; uncertain ones go to optional embedding and AI checks. Undecided requests: allow or block decides what happens when none of them can decide. A blocked request gets 400 prompt_rejected with no detail about why.
• Caller task hints. How far to trust a request's task_type: Trust hints, Same or a cheaper tier only, or Ignore hints; always classify. Any API key can send a hint, so trusting them lets a caller pick the premium tier.
• AI task classifier. When keyword rules are not confident, sends the latest user message (its first 800 characters) to the cheapest model you have configured, with your own keys, possibly at another provider. Each call is metered as an internal_classification decision. Off by default; without it, unmatched requests get the default mid-tier classification.
• Shadow recommendations. For requests on a non-tiered policy, logs which model tiered routing would have chosen without changing what runs. Results appear under Improve › Routing › Shadow recommendations. It can add classifier latency, and classifier cost if the AI classifier is on.

Limits and budgets

Rate and concurrency limits. On Gateway › Runtime choose Add limit, pick Applies to (Whole workspace or one API key) and set Max per minute (1 to 100,000) and Max at once (1 to 10,000). Limits are opt-in. A request needs room under both its workspace limit and its key limit; otherwise it gets 429 rate_limit_exceeded with the scope. The table shows each limit's count this minute and requests in flight now, with Save and Remove. Batch submissions count against the same limits.

Per-request and per-run ceilings. Set Per request, at most and Per run, at most under Settings › General › Defaults. The per-request ceiling applies to every routed request (see Cost ceilings above). The per-run ceiling caps the run_budget_usd a caller may set.

Monthly budgets. A workspace-wide budget under Governance › Budgets with enforcement Enforced: router traffic only, quality-gated downgrade or reject is enforced by the router: it reserves each request's worst-case cost against the budget before the call. A request that does not fit is refused with 402 monthly_budget_exceeded, unless its policy has a cheaper fallback and Improvements has a quality-approved model downgrade for that pair, in which case the fallback serves it (monthly_budget_downgrade). Tiered requests are never downgraded. See Governance for budgets and governance policies. Gateway › Runtime › Budget enforcement shows each run and monthly budget's ceiling, use and reservations.

Gemini prompts are counted with an approximate tokenizer, so the router pads them by 20% for these checks. The bill always uses the provider's own usage.

Caching

Everything under Gateway › Caching is off until you turn it on, stored encrypted and scoped to your workspace. Save with Add cache settings.

Exact response cache. Reuses an identical earlier answer for 15 minutes. A request is eligible only when it is non-streaming, sets temperature to 0, sends no tools, tool_choice, response_format or images, is not a /v1/messages request, is not marked sensitive: true or no-store, and its action does not contain a word such as auth, credentials, secret, payment, health, medical, legal, hr, personnel, security or pii. The match covers the provider, model and whole request including customer_ref, so one customer never gets another's answer; trace_id, action, feature, team and environment are left out. Responses over 256 KB are not stored, and an answer served by a fallback or downgrade model is never stored under the requested model.

Semantic response cache. Experimental. Reuses an answer when the last user message is similar enough and everything else (the system prompt, earlier turns and every parameter) matches exactly. On top of the exact cache's rules, the request must have plain text messages only, at most 64 of them and 12,000 bytes of text, ending with a user message. Similarity is computed with OpenAI's text-embedding-3-small using your OpenAI inference key, so it needs one; each embedding is metered as an internal_embedding decision. An embedding is made only when there are stored entries to compare against. Set Similarity from 0.900 to 0.999. Entries last 24 hours, up to 1,000 per workspace, and responses over 64 KB are not stored.

A cache hit makes no provider call and records no usage. The response carries x-tokenmeter-cache: hit or semantic-hit, and the decision's reason is response_cache_hit or semantic_response_cache_hit. Hits are counted on Improve › Routing and the Gateway overview.

Idempotent replay. Keeps each completed non-streaming response encrypted for 24 hours, so a retry carrying the same X-TokenMeter-Idempotency-Key and the same request gets it back without a second call or charge. Streamed responses, responses withheld by a guardrail and bodies over 256 KiB are not kept. See Router for the other outcomes of a reused key.

Prompt and output capture. Stores the message array (Prompts) and the non-streaming answer (Outputs) of router requests, encrypted, for Keep 1 to 30 days. Streaming, /v1/messages, rejected requests, SDK and OpenTelemetry traffic are excluded. Captured content is not viewable in the dashboard; Gateway › Runtime › Content retention shows how many items are held and when the next expires.

Agent Recall

Recall keeps long agent runs smaller: large tool results from earlier turns are archived and replaced by a short marker, and the model can fetch one back when it needs the exact text.

1. Under Gateway › Caching › Agent Recall, turn it on and set Keep (1 to 30 days), Minimum (1,024 to 1,000,000 bytes) and Allowed tools, a comma-separated list of tool names. At least one tool name is required.
2. In your agent, send recall with enabled: true and a session_id that is stable for the run, and include the tokenmeter_recall tool in tools. The TypeScript SDK's RecallClient does both and runs the tool; see the API reference.

The router then replaces each tool-result message whose name is on the allowed list and whose text is at least the minimum (the larger of the workspace's and the request's, up to 1,000,000 bytes) with a marker holding a Recall ID. When the model calls tokenmeter_recall, your loop fetches the original with POST /v1/recall/retrieve (recall_id and session_id), using the same ingest key that sent it. Requests without the tokenmeter_recall tool are never changed, and if storage fails the original text is kept. Improve › Routing › Token savers shows Recall: net tokens avoided.

Batch routing and the savings baseline

Async at batch rates (Gateway › Defaults). When on, a request sent with Prefer: respond-async is queued through the provider's batch API and answered within 24 hours, usually at the provider's batch price. Only OpenAI and Anthropic models qualify, and requests that stream, send tools or images, or use run or cost budget fields are refused. Ordinary requests are never moved to batches. Queued batches are listed under Improve › Routing › Async batches. Explicit POST /v1/batches submissions do not need this switch; see Router.

Savings measured against (Gateway › Defaults). The model routing savings are priced against when the router chose the model (automatic tiering) or the caller sent a placeholder model name: auto, default or tokenmeter-auto. A request that names a concrete model is measured against that model. Choose one of your capability models, or Not declared (unpriced). Changing it restates past savings figures. A placeholder name needs a policy that routes it, since the router otherwise sends the name to the provider as it is.

Experiments and model evaluations

Evaluate › Experiments has three tabs. Each comparison starts as a draft and serves traffic only after an admin approves it.

Prompt experiments

Compare two prompt revisions of your own on a small share of one workload. Token Meter never stores or rewrites prompt content; your application builds the prompt.

1. New experiment: name, provider, the exact selected model, an exact action and/or feature, the control and candidate variant IDs and revisions, Candidate allocation (1 to 20%), minimum requests, success outcomes and quality outcomes per variant, and the largest success-rate, quality and latency regressions you accept. Save draft.
2. Approve and start. Assignment runs for up to 14 days, followed by a 7-day window for outcomes to arrive. Pause stops it.
3. Before building each prompt, your application asks the router for its variant with POST /v1/prompt-experiments/assignment, sending experiment_id, cohort_key (the stable unit you assign, such as a user), action, feature, model and provider. The answer has variant_id and revision.
4. Send the routed request with all four headers X-TokenMeter-Prompt-Experiment-Id, X-TokenMeter-Prompt-Cohort-Key, X-TokenMeter-Prompt-Variant-Id and X-TokenMeter-Prompt-Revision. Sending only some is refused (prompt_experiment_headers_incomplete), and a variant that does not match the assignment gets 409 prompt_experiment_assignment_mismatch.
5. Report success (and, when its threshold is not zero, quality) outcomes to the ingest API with POST /v1/outcomes:batch. Quality outcomes use outcome_kind quality with quality_score from 0 to 100.

{
  "events": [
    { "outcome_kind": "success", "success": true, "routing_decision_id": "<from x-tokenmeter-routing-decision>" }
  ]
}

Only one running prompt experiment may cover a workload.

Model evaluations

Measure a candidate model against a baseline on your real traffic before it serves anyone.

1. Evaluation settings: tick Allow sampled model evaluation and its short-lived encrypted work queue, and to build a replay set also Also allow encrypted prompt and response capture for 30-day offline replay. Save evaluation setting.
2. New campaign: Baseline model, Candidate model, an optional Rubric judge (an OpenAI or Anthropic model from a different family than both), Evaluation sample rate (0 to 1), Daily candidate cap (USD) and Live rollout (%) (keep 0 to stay off production traffic). Create draft campaign. All models need a capability record.
3. While the campaign is a draft and capture is on, the router keeps encrypted samples of the baseline's matching traffic for replay (non-streaming requests without tools or images, not /v1/messages, up to 256 KB each).
4. Queue replay evaluation runs the samples against the candidate (and judge) in the background. The run passes only with at least 300 items, at most 3% judged worse, at most 0.2% critical failures, at least 99% format passes, at least 20% cost saving and at most 10% latency regression. Open the campaign for its status, the reasons it failed and the Evaluation report (Markdown).
5. Activate appears once the run has passed. An active campaign samples matching traffic at the sample rate: after the customer has their answer, the candidate (and judge) answer the same request in the background, each metered as its own decision, until the daily cap is spent. Repeated candidate errors pause the campaign. Live % serves that share of matching traffic from the candidate instead (reason evaluation_canary); Update changes it and Pause stops the campaign.

Live model experiments

Serve a share of one workload from a candidate model and compare it with the model it would otherwise get.

1. New experiment: name, an exact action and/or feature, an optional requested model, the Control model (served today) and Treatment model, Treatment allocation (1 to 50%), minimum decisions and linked success outcomes per arm, and the allowed cost, failure-rate, p95 latency and quality changes. Both models need an enabled capability record. Save draft.
2. Approve and start. It is refused while another running experiment covers some of the same traffic, or an Automatic tiering policy serves it. It runs for 14 days.
3. Only requests that carry a stable key (X-TokenMeter-Rollout-Key, customer_ref, user or trace_id) and would otherwise be served by the control model are enrolled. A failed treatment request falls back to the control model.
4. The evaluator measures both arms every 15 minutes: decisions, failure rate, p95 latency, cost per request and success rate. Collecting, Within tolerances, Breach or Cost not provable. A breach pauses the experiment automatically.
5. Once every tolerance passes at the minimum samples it shows Ready to promote. Promote creates a routing policy for the same scope that targets the treatment model, falls back to the control model and starts at 5% rollout; ramp it from Gateway › Router. Roll back ends the experiment without changing routing.

Results are fixed tolerances at minimum samples, not a significance test.

Savings

Improve › Routing shows, for 24 hours, 7 or 30 days:

• Routed calls, exact and semantic cache hits, completed, fallbacks, rejected or failed, and settlement pending.
• The latest 100 decisions, sortable by time, latency and result.
• Routing savings: the same tokens priced on the model each request would otherwise have used and on the model that served it, at your rates. Fallbacks, budget downgrades and evaluation canary traffic are listed separately and never counted as savings. The calculation is explained in Routing savings methodology.
• Token savers: tokens removed by Logs, decisions on Quiet policies, and Recall's net tokens avoided, as token estimates rather than dollars.
• Shadow recommendations: per task type, the model tiered routing would have picked and its estimated cost, when shadow recommendations are on.

The Gateway overview shows the same estimate as Routing estimate, next to Verified savings (realized savings from Improvements this month).

LiteLLM

To route through your own LiteLLM proxy, use the three LiteLLM tabs on Gateway › Connect:

1. LiteLLM credential: the Gateway URL (its origin, optionally ending in /v1) and a LiteLLM virtual key. The URL's origin must be on your deployment's allowlist (ROUTING_LITELLM_ALLOWED_ORIGINS); otherwise saving fails with an error naming that allowlist.
2. LiteLLM capability: approve each model by its catalog id, for example openai/gpt-4o, with the same limits and flags as other models. Prices come from a weekly catalog job; until it has synced, LiteLLM routes have no list prices and the tab says so.
3. Route through LiteLLM: Target catalog model, optional Requested model match, Action match and Feature match, Priority and Rollout percent. This creates an ordinary routing policy; Retire removes it.

LiteLLM supports POST /v1/batches but not Prefer: respond-async.

Limits

• A request body can be at most 2 MB; a batch at most 1,000 requests.
• /v1/messages requests stay on Anthropic models and are not cached, captured or shadow-evaluated.
• Google Gemini has no batch or async support.
• Timeouts and the circuit breaker's thresholds are fixed.
• Captured prompts, outputs, replay samples and Recall items are never viewable in the dashboard.

Troubleshooting