Router

Before the first request

A workspace admin sets up two things under Gateway › Connect. Without them the router refuses the call.

1. An inference key for each provider you call: OpenAI, Anthropic or Google Gemini (or a LiteLLM gateway). Without one the router answers 424 provider_not_configured.
2. A model capability record for every model you call: its context and output limits and whether it supports tools, structured output and vision. Without one the router answers 400 model_capability_not_configured. A request that sends tools to a model saved without Tools gets 400 model_does_not_support_tools.

Which provider serves a request

With no routing policy, a model whose name starts with claude- goes to Anthropic and every other model goes to OpenAI. To send traffic to Google Gemini or through a LiteLLM gateway, create a routing policy that targets that model (Gateway › Router). A policy can also move a request to a different model, which is how cheaper-model routing works.

Environment only

The OpenAI SDKs read these, so many apps need no code change:

OPENAI_BASE_URL=https://router.example.com/v1
OPENAI_API_KEY=$TOKEN_METER_KEY

In code

import OpenAI from "openai";

const openai = new OpenAI({ baseURL: "https://router.example.com/v1", apiKey: process.env.TOKEN_METER_KEY });
await openai.chat.completions.create({ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello" }] });

The key can be sent as Authorization: Bearer <your ingest key> or as an x-api-key header.

Anthropic Messages API

Point the Anthropic SDK at the router's origin (the SDK adds /v1/messages itself):

import os
from anthropic import Anthropic

client = Anthropic(base_url="https://router.example.com", api_key=os.environ["TOKEN_METER_KEY"])
client.messages.create(model="claude-haiku-4-5", max_tokens=512, messages=[{"role": "user", "content": "Hello"}])

• Requests to /v1/messages can only be served by Anthropic models. A routing policy that targets another provider makes them fail with 400 native_anthropic_cross_provider_routing_not_enabled; fallbacks and experiments that point at another provider are skipped for them.
• Tools, system, thinking, metadata and the cache_control markers inside messages are passed through.
• /v1/messages requests are never answered from the router's response caches, and are not captured or sampled for shadow model comparisons.

A claude- model called through /v1/chat/completions is translated to Anthropic's format, but only for plain string message content. Tools, tool_choice and response_format are refused there (anthropic_tool_and_structured_output_translation_not_enabled); use /v1/messages for those.

Attribution

Send the tags as top-level fields of the request body. The router records them and removes them before forwarding to the provider.

// Built as a variable, so TypeScript accepts the extra fields the OpenAI types don't list.
const body = {
  model: "gpt-4o-mini",
  messages: [{ role: "user" as const, content: "Hello" }],
  action: "generate-report",
  customer_ref: "acme",
  feature: "reports",
};
await openai.chat.completions.create(body);

action, feature and the requested model are also what routing policies match on.

Other fields the router reads

These are optional, read by the router and never forwarded to the provider:

Any other field the router does not know is refused with 400 unsupported_request_field:<name>, so nothing that can raise cost reaches the provider unaccounted for. On /v1/chat/completions the router accepts messages, max_tokens, max_completion_tokens, temperature, top_p, tools, tool_choice, response_format, stop, user, parallel_tool_calls, stream, stream_options, seed, presence_penalty, frequency_penalty, logit_bias, logprobs, top_logprobs, reasoning_effort and prompt_cache_key. n and service_tier are refused.

Streaming works on both endpoints. The router asks the provider for usage on every stream itself, so whatever you send in stream_options is replaced.

Headers

Reusing an idempotency key:

• Same request, first one completed, and Idempotent replay on (Gateway › Caching): the stored response comes back with x-tokenmeter-idempotent-replay: true and no second charge.
• Same key, different request: 422 idempotency_key_reused_with_different_request.
• First request still running: 409 idempotency_request_in_progress with Retry-After: 2.
• Otherwise: 409 idempotency_key_already_used.

Responses

Every routed response carries x-tokenmeter-routing-decision: <decision id>. Answers served from a cache also carry x-tokenmeter-cache: hit (exact) or semantic-hit with x-tokenmeter-cache-similarity.

GET /v1/routing-decisions/<decision id> (same key) returns that decision's status, selected_provider, selected_model, estimated_max_cost_usd, created_at and completed_at. A streamed response can finish an instant before its decision is marked completed, so poll again briefly if it still reads reserved.

A request with Prefer: respond-async gets 202 with the batch id, poll_url and items_url. Read progress at GET /v1/batches/<id> and the answers at GET /v1/batches/<id>/items. It is refused (400 request_not_eligible_for_batch_arbitrage) when it streams, sends tools or images, uses run_id, run_budget_usd or max_cost_usd, or is served by a provider other than OpenAI or Anthropic.

Errors

Router errors are JSON {"error": "<code>"}. A provider's own error is passed through with the provider's status and body.

Batches

POST /v1/batches submits up to 1,000 requests to one model as a single provider batch:

{
  "model": "gpt-4o-mini",
  "action": "nightly-summaries",
  "requests": [
    { "id": "item-1", "input": { "messages": [{ "role": "user", "content": "Summarize ..." }], "customer_ref": "acme" } }
  ]
}

provider is optional (openai, anthropic or litellm; Gemini has no batch transport). action and feature at the top level pick the routing policy for the whole batch; attribution fields go in each item's input. Each item's id is your own and must be unique in the batch; items cannot stream or send tools or images. The whole batch is checked against rate limits, cost caps, budgets and governance policies before it is accepted with 202. Poll it with the two GET /v1/batches/... routes above. Send X-TokenMeter-Idempotency-Key to make a retried submission return the batch it already created.

Coding tools: Cline and Continue

Both call models with your own API keys, so the router can meter them without any plugin.

Cline: in Cline's settings, choose the API provider OpenAI Compatible, set the base URL to https://router.example.com/v1, the API key to your Token Meter ingest key, and the model ID to a model your router credentials can reach (for example gpt-4o-mini).

Continue: add a model to config.yaml that points at the router:

models:
  - name: GPT-4o mini (metered)
    provider: openai
    model: gpt-4o-mini
    apiBase: https://router.example.com/v1
    apiKey: <your Token Meter ingest key>

Calls then show on the dashboard with exact tokens and cost. Neither tool sends the attribution fields above, so these calls arrive without a customer, feature or team.

SDK or router, not both

A call sent through the router and also made with a wrapped client is metered twice. Use one per call path.