Duaer

Guides

2027 LLM Inference Cost per Token: Why Price Sheets Mislead You

This article from the Duaer documentation site explains why the listed price per token is not the real unit cost of enterprise AI, and how to calculate the cost of each accepted outcome using your own traffic.

The Numbers on the Price Sheet Are Not Your Unit Cost

Most teams first encounter LLM inference cost per token in a procurement meeting. The vendor quotes a price per million input tokens and per million output tokens, the finance lead multiplies it by estimated monthly usage, and the project gets a budget. Three months later the bill is two or three times the estimate, or the feature works but nobody can say whether it is worth the spend. The problem is rarely the list price itself. It is that the team is measuring the wrong thing. The price per token is one input to cost, not the cost itself.

This article explains where the gap comes from, how to calculate a per-token cost that reflects real traffic, and how to connect that number to business results. If you are still estimating the total budget for adopting AI across the company, see our guide: How to Calculate Enterprise AI Adoption Costs Beyond Model Pricing. Here we focus only on unit cost.

Why the List Price Rarely Equals What You Actually Spend

Several factors shape the effective cost per token, and the price sheet shows none of them.

First, input and output tokens are priced differently, and the ratio between them in your product is often not what the vendor assumed in the demo. Retrieval-augmented systems typically send long context and get short answers, so input dominates. Agents that plan, call tools, and summarize results can generate far more output than a simple chat assistant. A blended rate calculated on a 1:1 ratio can be off in either direction.

Second, context grows quietly. A prompt template that started at a few hundred tokens picks up system instructions, compliance text, retrieved passages, and conversation history over several months. Nobody decides to double the prompt. It accumulates. Every added block raises the cost of every call, and unless you log token counts per request, the growth is almost invisible.

Third, some models consume tokens you never see. Reasoning-capable models may generate internal tokens before they produce the visible answer, and those tokens are usually billed as output. A low unit price does not mean a cheap answer.

Fourth, retries and fallbacks multiply calls. A failed JSON parse, a timeout, or a validation step that asks the model to try again all consume tokens. The true cost of one successful result is the sum of every attempt that led to it.

Calculate Cost per Token From Your Own Traffic, Not Benchmark Data

A usable calculation must start from real logs. For API-based models, the basic formula is:

Cost per request = (input tokens × input price) + (output tokens × output price), summed across all calls, including retries and tool-calling rounds.

Then, for each workflow, use the token mix from your logs rather than the vendor's example to calculate a blended cost per 1,000 tokens. Calculate it per feature, because a summarization job and a customer-facing chat behave very differently. Averaging across the whole company hides where the money actually goes.

For self-hosted models, the method differs. Cost per token is roughly the hourly cost of accelerators and supporting infrastructure, divided by the number of tokens actually produced in that hour. The word actually matters. If traffic is bursty and GPU utilization is low, the cost per token will be high even when the hardware looks cheap on paper. Batch size, concurrency, and request length all affect this number. If a team compares API per-token prices with GPU rental quotes without modeling utilization, it is comparing two different things. A Breakdown of Private LLM Deployment Costs explains why hardware quotes need to be converted into operating-level figures before a fair comparison is possible.

Common Mistakes That Push Unit Cost Up

The first mistake is measuring tokens at the model boundary rather than the application boundary. Your application may add system prompts, retrieval steps, and formatting calls. What matters to the business is the total across all of these.

The second mistake is optimizing the price per token while ignoring the number of tokens each task consumes. Switching to a cheaper model can raise total cost if it needs two extra rounds to reach the same answer. Compare the cost per successful task, not the cost per call.

The third mistake is treating caching as a free benefit. For workloads with stable system prompts, prompt caching and reuse of repeated prefixes can reduce costs significantly, but only if the prompt structure stays stable. Reordering instructions or inserting a timestamp at the top can quietly eliminate the savings.

The fourth mistake is forgetting the tokens consumed by evaluation. Regression tests, LLM-as-judge scoring, and shadow traffic all use tokens. In mature teams this can be a meaningful line item, and it is often missing from the original budget.

Tie Unit Cost to Business Outcomes and Build a Simple Measurement Loop

A cost per token means little unless it is tied to the results the business actually cares about. For each workflow, choose one outcome metric: for example, a resolved support ticket, a contract clause flagged for review, a published product description, or a document summary accepted by a reviewer. Then calculate the cost per outcome, including retries, the reviewer time that follows, and the share of outputs that get rejected.

This step often reveals the real economics of a pilot. A feature with a very low cost per call can still lose money if half its outputs need rework. Conversely, a more expensive model that removes a review step may be the cheaper choice. That is why the unit to track is the cost per accepted outcome, with cost per token as only one component.

You do not need a large platform to start. For each model call, record the workflow name, model, input tokens, output tokens, whether it was a retry, and whether the result was accepted. Summarize weekly and focus on three numbers: cost per accepted outcome, the average input-to-output ratio, and the share of spend going to retries. When any of these shifts noticeably, investigate prompts, retrieval steps, or routing logic before blaming the model vendor.

Set thresholds for each workflow and decide in advance what happens when they are crossed. Options include shortening context, routing simple requests to a smaller model, caching stable prefixes, or pausing the feature altogether. Agreeing on responses ahead of time prevents the common pattern of discovering the bill, arguing about it, and then doing nothing for a quarter.

What You Can Do This Week

Pick one AI feature that is already running. Export 30 days of request logs, or ask your platform team for token statistics. Calculate the blended cost per 1,000 tokens for that feature, then divide total monthly spend by the number of accepted outcomes. Write both numbers on one page and place them next to the vendor's list price. The gap between them is the starting point for every optimization decision that follows.

If you would rather not build this logging and reporting by hand, you can configure the workflow in Duaer (https://www.duaer.com), organize usage tracking by workflow, and view outcome-level costs side by side. This is a practical way to keep the measurement loop running, but it cannot replace the judgment about which outcomes your business actually values. Sign up at https://www.duaer.com to try it with one of your own workflows.

If you tell us which workflow you most want to measure first, we can provide a written rollout plan covering how to instrument your calls, which metrics to report, and where costs tend to hide.

https://api.duaer.com/form/c59a907a-3697-4edf-a04d-3ea0fb72fdc8

The price per token tells you what the vendor charges. The cost per accepted outcome tells you what the business actually pays. The second number is the one that should decide whether an AI feature keeps its budget.

Questions

How do you calculate LLM inference cost per token for an enterprise?

Calculate it from real request logs. For each call, multiply input tokens by the input price, add output tokens multiplied by the output price, and sum across all calls, including retries and tool-calling rounds. Then calculate a blended cost per 1,000 tokens for each feature separately, so that a company-wide average does not hide the true cost distribution.

Why is the API bill often much higher than the budget?

Common causes include an actual input-to-output ratio that differs from the estimate, prompt context that grows over time, internal reasoning tokens that are billed as output, failed requests, timeouts and validation retries that increase the number of calls, and evaluation or regression testing whose token usage was never included in the budget. Logging token counts for every call is the foundation for finding the cause.

How should you calculate cost per token for a self-hosted model?

It is roughly the hourly cost of the accelerators and related infrastructure, divided by the number of tokens actually produced in that hour. Because utilization has a direct effect on the result, if traffic is bursty you should model actual utilization rather than using the hardware's theoretical throughput. A fair comparison with API pricing only works when both are expressed on the same operating basis.