# Router

> Meter OpenAI-compatible and Anthropic Messages traffic by changing a base URL instead of code. Team workspaces only.

The router speaks the OpenAI Chat Completions API at `/v1/chat/completions` and the Anthropic Messages API at `/v1/messages`. Point a client at it with your Token Meter ingest key as the API key; it forwards each request with your own provider key (stored encrypted in the dashboard under Gateway › Connect) and meters it. It can also route to cheaper models, cache, and enforce budgets, all configured in the dashboard: see [Gateway and routing](/docs/gateway.md).

The router's address is on the dashboard's setup page (Connect › Router). The examples below use `https://router.example.com`.

## Before the first request

A workspace admin sets up two things under **Gateway › Connect**. Without them the router refuses the call.

1. **An inference key for each provider you call:** OpenAI, Anthropic or Google Gemini (or a LiteLLM gateway). Without one the router answers `424 provider_not_configured`.
2. **A model capability record for every model you call:** its context and output limits and whether it supports tools, structured output and vision. Without one the router answers `400 model_capability_not_configured`. A request that sends `tools` to a model saved without Tools gets `400 model_does_not_support_tools`.

## Which provider serves a request

With no routing policy, a model whose name starts with `claude-` goes to Anthropic and every other model goes to OpenAI. To send traffic to Google Gemini or through a LiteLLM gateway, create a routing policy that targets that model (Gateway › Router). A policy can also move a request to a different model, which is how cheaper-model routing works.

## Environment only

The OpenAI SDKs read these, so many apps need no code change:

```shell
OPENAI_BASE_URL=https://router.example.com/v1
OPENAI_API_KEY=$TOKEN_METER_KEY
```

## In code

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://router.example.com/v1", api_key=os.environ["TOKEN_METER_KEY"])
client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello"}])
```

```typescript
import OpenAI from "openai";

const openai = new OpenAI({ baseURL: "https://router.example.com/v1", apiKey: process.env.TOKEN_METER_KEY });
await openai.chat.completions.create({ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello" }] });
```

The key can be sent as `Authorization: Bearer <your ingest key>` or as an `x-api-key` header.

## Anthropic Messages API

Point the Anthropic SDK at the router's origin (the SDK adds `/v1/messages` itself):

```python
import os
from anthropic import Anthropic

client = Anthropic(base_url="https://router.example.com", api_key=os.environ["TOKEN_METER_KEY"])
client.messages.create(model="claude-haiku-4-5", max_tokens=512, messages=[{"role": "user", "content": "Hello"}])
```

- Requests to `/v1/messages` can only be served by Anthropic models. A routing policy that targets another provider makes them fail with `400 native_anthropic_cross_provider_routing_not_enabled`; fallbacks and experiments that point at another provider are skipped for them.
- Tools, `system`, `thinking`, `metadata` and the `cache_control` markers inside messages are passed through.
- `/v1/messages` requests are never answered from the router's response caches, and are not captured or sampled for shadow model comparisons.

A `claude-` model called through `/v1/chat/completions` is translated to Anthropic's format, but only for plain string message content. Tools, `tool_choice` and `response_format` are refused there (`anthropic_tool_and_structured_output_translation_not_enabled`); use `/v1/messages` for those.

## Attribution

Send the tags as top-level fields of the request body. The router records them and removes them before forwarding to the provider.

| Field | Meaning |
|---|---|
| `action` | What the call does, e.g. `"generate-report"`. |
| `customer_ref` | The end customer or tenant (an opaque id). |
| `feature` | The product surface. |
| `team` | The owning team. |
| `environment` | e.g. `"prod"`. |
| `trace_id` | Optional, to group calls. |

```python
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"action": "generate-report", "customer_ref": "acme", "feature": "reports"},
)
```

```typescript
// Built as a variable, so TypeScript accepts the extra fields the OpenAI types don't list.
const body = {
  model: "gpt-4o-mini",
  messages: [{ role: "user" as const, content: "Hello" }],
  action: "generate-report",
  customer_ref: "acme",
  feature: "reports",
};
await openai.chat.completions.create(body);
```

`action`, `feature` and the requested model are also what routing policies match on.

## Other fields the router reads

These are optional, read by the router and never forwarded to the provider:

| Field | Effect |
|---|---|
| `max_cost_usd` | Refuse the request (`402 estimated_cost_exceeds_max_cost_usd`) when its worst-case cost is above this amount. |
| `run_id` | Groups calls into one run with a shared budget. Requires `max_cost_usd`. Up to 200 characters. |
| `run_budget_usd` | The run's total budget. The first call of a run must send it; later values are ignored. Only with `run_id`. A request that does not fit in what is left gets `402 run_budget_exceeded`. |
| `task_type` | A task hint for automatic tiering. How far it is trusted is set under Gateway › Router › Request handling. |
| `data_classification` | Data classes you declare (a string or a list), added to the ones the router detects in the text and checked against governance policies that block data classes. |
| `sensitive` | `true` keeps the request out of the response caches. |
| `cache_control` | `"no-store"` keeps the request out of the response caches. |
| `recall` | Agent Recall settings; see the [API reference](/docs/reference.md). |

Any other field the router does not know is refused with `400 unsupported_request_field:<name>`, so nothing that can raise cost reaches the provider unaccounted for. On `/v1/chat/completions` the router accepts `messages`, `max_tokens`, `max_completion_tokens`, `temperature`, `top_p`, `tools`, `tool_choice`, `response_format`, `stop`, `user`, `parallel_tool_calls`, `stream`, `stream_options`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`, `top_logprobs`, `reasoning_effort` and `prompt_cache_key`. `n` and `service_tier` are refused.

Streaming works on both endpoints. The router asks the provider for usage on every stream itself, so whatever you send in `stream_options` is replaced.

## Headers

| Header | Effect |
|---|---|
| `X-TokenMeter-Rollout-Key` | A stable key (up to 200 characters) that keeps one user or conversation on the same side of a partial policy rollout or experiment. Without it the router uses `customer_ref`, then `user`, then `trace_id`. |
| `X-TokenMeter-Idempotency-Key` | A UUID. A retry with the same key never makes a second provider call: see below. Any other value is refused with `400 invalid_idempotency_key`. |
| `Prefer: respond-async` | Queue the request through the provider's batch API at batch prices instead of answering now. Needs **Async at batch rates** on under Gateway › Defaults. |
| `Cache-Control: no-store` | Keep this request out of the response caches. |

Reusing an idempotency key:

- Same request, first one completed, and **Idempotent replay** on (Gateway › Caching): the stored response comes back with `x-tokenmeter-idempotent-replay: true` and no second charge.
- Same key, different request: `422 idempotency_key_reused_with_different_request`.
- First request still running: `409 idempotency_request_in_progress` with `Retry-After: 2`.
- Otherwise: `409 idempotency_key_already_used`.

## Responses

Every routed response carries `x-tokenmeter-routing-decision: <decision id>`. Answers served from a cache also carry `x-tokenmeter-cache: hit` (exact) or `semantic-hit` with `x-tokenmeter-cache-similarity`.

`GET /v1/routing-decisions/<decision id>` (same key) returns that decision's `status`, `selected_provider`, `selected_model`, `estimated_max_cost_usd`, `created_at` and `completed_at`. A streamed response can finish an instant before its decision is marked completed, so poll again briefly if it still reads `reserved`.

A request with `Prefer: respond-async` gets `202` with the batch `id`, `poll_url` and `items_url`. Read progress at `GET /v1/batches/<id>` and the answers at `GET /v1/batches/<id>/items`. It is refused (`400 request_not_eligible_for_batch_arbitrage`) when it streams, sends tools or images, uses `run_id`, `run_budget_usd` or `max_cost_usd`, or is served by a provider other than OpenAI or Anthropic.

## Errors

Router errors are JSON `{"error": "<code>"}`. A provider's own error is passed through with the provider's status and body.

| Status | Codes |
|---|---|
| 400 | `model_required`, `invalid_json`, `unsupported_request_field:<name>`, `model_capability_not_configured`, `model_does_not_support_tools` (or `_structured_output`, `_vision`), `max_tokens_exceeds_model_capability`, `request_exceeds_model_context_capability`, `prompt_rejected` (input guardrail), `response_rejected` (output withheld, `billed: false`), `tiered_routing_not_configured`, `model_blocked_by_governance_policy`, `request_blocked_by_governance_policy` |
| 401 | `invalid_key` |
| 402 | `org_max_request_cost_usd_exceeded`, `policy_max_request_cost_usd_exceeded`, `estimated_cost_exceeds_max_cost_usd`, `monthly_budget_exceeded`, `run_budget_exceeded` (each with the estimate and the limit) |
| 403 | `batch_arbitrage_not_enabled` |
| 424 | `provider_not_configured`: no inference key for the provider that would serve the request |
| 429 | `rate_limit_exceeded` with `scope` `org` or `key` |
| 502 | `provider_network_failure`: the provider could not be reached |
| 503 | `provider_circuit_open` (with `Retry-After: 30`), `governance_policies_unavailable`, `monthly_budget_unavailable` |

## Batches

`POST /v1/batches` submits up to 1,000 requests to one model as a single provider batch:

```json
{
  "model": "gpt-4o-mini",
  "action": "nightly-summaries",
  "requests": [
    { "id": "item-1", "input": { "messages": [{ "role": "user", "content": "Summarize ..." }], "customer_ref": "acme" } }
  ]
}
```

`provider` is optional (`openai`, `anthropic` or `litellm`; Gemini has no batch transport). `action` and `feature` at the top level pick the routing policy for the whole batch; attribution fields go in each item's `input`. Each item's `id` is your own and must be unique in the batch; items cannot stream or send tools or images. The whole batch is checked against rate limits, cost caps, budgets and governance policies before it is accepted with `202`. Poll it with the two `GET /v1/batches/...` routes above. Send `X-TokenMeter-Idempotency-Key` to make a retried submission return the batch it already created.

## Coding tools: Cline and Continue

Both call models with your own API keys, so the router can meter them without any plugin.

**Cline:** in Cline's settings, choose the API provider **OpenAI Compatible**, set the base URL to `https://router.example.com/v1`, the API key to your Token Meter ingest key, and the model ID to a model your router credentials can reach (for example `gpt-4o-mini`).

**Continue:** add a model to `config.yaml` that points at the router:

```yaml
models:
  - name: GPT-4o mini (metered)
    provider: openai
    model: gpt-4o-mini
    apiBase: https://router.example.com/v1
    apiKey: <your Token Meter ingest key>
```

Calls then show on the dashboard with exact tokens and cost. Neither tool sends the attribution fields above, so these calls arrive without a customer, feature or team. If you also send Cline's OpenTelemetry logs or Continue's data events to Token Meter, the same calls count twice: pick one route per tool.

## SDK or router, not both

A call sent through the router and also made with a wrapped client is metered twice. Use one per call path.