Guides

Guide

How to reduce LLM API costs: measure first, then cut

Updated

To cut an LLM bill you first need to know which actions, features and customers drive it. This guide shows how to find out, which levers lower the cost, what each one costs you, and how to measure the saving without fooling yourself.

What makes an LLM API bill grow?

An LLM API bill is the tokens each call uses, priced at the rate of the model that served it, summed over every call. So there are four ways to lower it: make fewer provider calls, use fewer tokens per call, serve the call from a cheaper model, or pay a lower rate for the same model. Limits then stop spend you never meant to make.

How do you find what is driving your LLM costs?

Tag every call with the action, customer, feature and team that caused it, when you make it. Provider invoices and usage exports have no customer or feature field, so this split can't be rebuilt afterwards. With tags, you can see what it costs to serve one customer, feature or team.

Token Meter takes the tags two ways: wrap your provider client with the SDK, which reads usage from the response and adds no network hop, or point an OpenAI or Anthropic SDK client at the router and send the tags as body fields. The router removes them before the call reaches the provider. A model you name runs on your own provider key, which an Owner or Admin adds under Router › Providers with a capability record for the model.

summarize.ts
import OpenAI from "openai";

// The router's address. The OpenAI SDK needs /v1 at the end.
const openai = new OpenAI({ baseURL: "https://router.timewastevc.com/v1", apiKey: process.env.TOKEN_METER_KEY });

// Built as a variable, so TypeScript accepts the extra fields the OpenAI types don't list.
const body = {
  model: "gpt-4o-mini",
  messages: [{ role: "user" as const, content: "Summarize ..." }],
  action: "summarize-ticket",
  customer_ref: "acme",
  feature: "support",
};
await openai.chat.completions.create(body);

Routing policies match on the same action and feature, so the tags you add to measure are the ones you later route by. Use the SDK or the router for a call path, not both, or it is metered twice.

Two alert rule kinds (paid plans) flag spend you weren't expecting: Provider spike compares today's spend with yesterday's, and Anomaly (runaway trace) fires when one trace makes far more calls or costs far more than usual for its action. See also LLM cost per customer and OpenAI cost tracking.

Which levers lower LLM API costs?

Token Meter's router applies five: route to a cheaper model, answer repeated requests from a cache, run work that can wait at batch prices, send fewer tokens, and cap spend.

Router traffic only

These levers act only on calls sent through the router. Usage metered by the SDKs, OpenTelemetry or billing imports is measured and alerted on, but not changed.

LeverWhat it changesWorks whenTrade-off
Cheaper-model routingThe model that serves the requestA cheaper model does the job for that action or featureQuality risk; test on your traffic first
Response cachingWhether the provider is called at allThe same non-streaming request at temperature 0 comes backNarrow eligibility; the semantic cache is experimental
Async batchesThe price you pay for the same modelThe answer can wait up to 24 hoursSlow; no Gemini, streaming, tools or images
Token saversTokens per callLong answers, long tool logs, long agent runsMeasured in tokens, not dollars
Limits and budgetsWhether an expensive request runsYou know the most a request, run or month should costRequests over the limit are refused

Only owners and admins configure the router and gateway. Setup and error codes: Router.

How do you route requests to a cheaper model without losing quality?

Test the cheaper model on your own traffic, then send a small share of matching requests to it and ramp up. A routing policy matches on requested model, action and feature, sends matching requests to a target model, and can fall back to another model when the target fails. On Growth and above, model: "auto" also picks a model per request, on Token Meter's keys up to US$25 a month per workspace: each task type starts on the strongest model and moves to a cheaper one only after that model passes judged sampled checks.

  1. Approve the models. Each provider needs an inference key and each model a capability record. The router refuses models without one.
  2. Compare on real traffic. A live model experiment serves 1 to 50% of one workload from the candidate for 14 days, measuring failure rate, p95 latency, cost per request and success rate; a breach of your tolerances pauses it. A model evaluation replays sampled traffic against the candidate before it serves anyone. Candidate and judge calls are real provider calls, metered like any other.
  3. Promote. Once every tolerance passes at the minimum sample, Promote creates a policy that targets the candidate, falls back to the current model and starts at 5% rollout.
  4. Ramp. Raise the rollout with Ramp on the policy's row under Router › Rules › Advanced › Policies. A new policy pauses itself if, in one hour with at least 50 decisions, its failure rate goes above 5% or its fallback rate above 10%.

Experiment results are fixed tolerances at minimum samples, not a significance test. Tiered routing goes further: it classifies each request by task and sends it to that task's model, assigned only after at least 200 independently judged paired responses. Details: Gateway and routing.

When does response caching reduce LLM costs?

When your app sends the same request more than once. A cache hit makes no provider call and records no usage. The exact cache reuses an identical answer for 15 minutes, only for non-streaming /v1/chat/completions requests at temperature 0 (or with no temperature, if you switch on Requests that set no temperature) with no tools, tool_choice, response_format or images. When a cache is on, an uncached response says why in x-tokenmeter-cache: ineligible:<reason>.

  • The match includes customer_ref, so one customer never gets another's answer. Requests marked sensitive, and actions with words such as auth, payment, health or pii, are never cached.
  • The semantic cache is experimental: it reuses an answer when the last user message is similar enough (you set 0.900 to 0.999) and everything else matches. Its embeddings use your OpenAI key and are metered.
  • Idempotent replay covers retries: the same request with the same X-TokenMeter-Idempotency-Key gets the stored response without a second call or charge.

All of these are off until you turn them on. Provider prompt caching, where the provider charges less for a repeated prompt prefix, is separate: the router passes Anthropic's cache_control markers through on /v1/messages and accepts OpenAI's prompt_cache_key.

How does a batch API lower LLM costs?

By trading speed for price. With Run non-urgent requests as cheaper batch jobs (results within 24 hours) switched on in Router defaults (the Defaults link on Router › Rules), a request sent with Prefer: respond-async is queued through the provider's batch API and answered within 24 hours, usually at the provider's batch price. Ordinary requests are never moved to batches.

Only OpenAI and Anthropic models qualify, and requests that stream, send tools or images, or use run or cost budget fields are refused. For bulk jobs, POST /v1/batches submits up to 1,000 requests to one named model as one provider batch, checked against the monthly request limit, rate limits, cost caps, budgets and governance policies before it is accepted. It doesn't need the switch, and it runs on your own provider key.

POST /v1/batches
{
  "model": "gpt-4o-mini",
  "action": "nightly-summaries",
  "requests": [
    { "id": "item-1", "input": { "messages": [{ "role": "user", "content": "Summarize ..." }], "customer_ref": "acme" } }
  ]
}

How do you send fewer tokens per call?

Turn on the token savers that fit. On a routing policy, Quiet asks for concise, answer-first responses and Logs shortens repetitive or oversized tool-result logs. Agent Recall replaces large tool results from earlier turns of an agent run with a short marker, and the model can ask for the exact text back.

Savings are reported in tokens, not dollars. Quiet adds an instruction to every request, and its benefit is not measured. Recall only changes requests that include the tokenmeter_recall tool.

How do you stop LLM costs from running away?

Put a ceiling on each request, each agent run and each month. Before calling the provider, the router works out a request's worst-case cost (the exact input tokens plus the output cap) and refuses it with HTTP 402 when that is over the limit.

  • Per request: a policy's Max request cost, the workspace's Per request, at most, or the caller's max_cost_usd.
  • Per run: run_id and run_budget_usd (with max_cost_usd on each call) give an agent's calls one shared budget.
  • Per month: an enforced workspace-wide budget (paid plans) refuses a routed request that doesn't fit, unless its policy has a cheaper fallback with a quality-approved downgrade for that model pair.
  • Everything else: budgets on a customer, feature, action, team, model or provider are checked every 60 seconds against month-to-date spend from every connected source, and alert at 50%, 80% and 100% by default and on a projected overrun. On the Free plan you can set one advisory, workspace-wide budget; scoped budgets, enforcement, alert rules and notification channels need a paid plan. See Governance and alerts.

How do you measure LLM cost savings honestly?

Price the same tokens twice, once on the model the request would otherwise have used and once on the model that served it, at your own rates. The difference is the saving. That is how Token Meter's routing savings figure works, and it is an estimate, not an invoice: the baseline model never ran.

DecisionIn the savings figure?
A policy, tiering or auto switched the modelCounted
A batch ran at the batch priceCounted, against the same model at the synchronous price
Fallbacks, budget downgrades, evaluation trafficShown separately, not counted
Cache hits, the router's internal calls, failed or rejected requestsNot counted
No baseline declared, no price, partial or unsettled usageUnpriced: left out of the dollars, shown in the coverage count

The figure is a range. A conversation moved to another model loses the prompt cache it had built, so the low end assumes the baseline would have read its whole prompt from cache, and the high end assumes it would not. It assumes the baseline would have used the same tokens. It does not measure quality: beside it, success rates are compared from the outcomes you report, and below 30 labelled switched decisions the page says the saving is unproven. Only a controlled comparison, such as an evaluation, shows a cheaper model is as good. Full method: routing savings methodology.

What should you do first?

Measure before you change anything, then start with the levers that leave the model alone: limits, batches for work that can wait, and exact caching. Switch models only after testing the cheaper one on your own traffic.

  1. Tag calls with action, customer and feature, and find the few that cost the most.
  2. Set a budget, and on a paid plan add alert channels and an anomaly rule for runaway traces.
  3. Move work that can wait, such as nightly summaries, to batches.
  4. Turn on the exact cache if you repeat temperature 0 requests.
  5. Test a cheaper model with an evaluation or a live experiment, then promote the experiment at 5% and ramp.
  6. Read the savings with their coverage and quality evidence, not the headline alone.

Start with the numbers

Create a workspace, add an ingest key and send one tagged call through the router or a wrapped client. You will see which actions cost the most before you change any routing.