AI token prices have fallen 280 times in two years. Enterprise AI bills have gone up 320% in the same period.

That gap is not a contradiction. It is the story of AI inference. And if your organisation is running AI in production, you are already living inside it.

The average enterprise AI budget grew from $1.2 million per year in 2024 to $7 million in 2026. 73% of organisations exceeded their AI cost projections. AI inference cost now represents 85% of enterprise AI budgets.

Most enterprise finance teams were not built to manage this. Most enterprise technology teams did not explain it clearly when they asked for the budget. This piece does.

What AI Inference Actually Is

Building an AI model and running an AI model are two completely different things. Understanding this distinction is the starting point for understanding why AI costs behave the way they do.

Training is the process of building the model. You feed it data. It learns patterns. This is expensive and happens once, or periodically when the model needs to be updated. The cost is large but bounded. You know what you are spending and when it ends.

Inference is the process of running the model. Every time a user submits a query, every time an AI agent takes a step in a workflow, every time a system asks the model to make a decision, inference happens. The model receives an input and generates a response. This is what happens in production, continuously, at scale.

Inference costs scale with usage. The more queries the system handles, the more it costs. The longer the inputs and outputs, the more it costs. The more capable the model, the more it costs per query.

This is fundamentally different from how enterprise software has historically been priced. Software licences are fixed costs. Infrastructure is largely predictable. AI inference is a consumption cost that scales with every user, every agent, and every workflow that touches it.

Uber's engineering organisation ran out of its entire annual AI coding budget two-thirds of the way through the year. Microsoft pulled back developer access to Claude Code for the same reason. Neither company is careless with money. Both were caught by the same math.

Why the Bill Keeps Rising Despite Falling Prices

This is the question that confuses most finance and technology leaders when they first encounter it.

The unit price of AI inference has fallen dramatically. A query that cost one dollar in 2023 costs a fraction of that today. Gartner forecasts that by 2030, running inference on a one-trillion-parameter model will cost providers more than 90% less than it did in 2025.

So why is the bill going up?

Because volume is growing faster than price is falling.

When enterprises move AI from pilot to production, the number of inference calls multiplies. A pilot with 50 users runs a few thousand queries per week. A production deployment with 5,000 users runs millions. When AI agents are added to the equation, the multiplication accelerates further. Each agent step is an inference call. A multi-step agent workflow generates dozens of inference calls per task. At enterprise scale, across multiple agents and workflows, the token volume is orders of magnitude larger than any pilot suggested.

The FinOps Foundation's 2026 State of FinOps report makes the structural reason clear. The share of FinOps teams managing AI spend rose from 31% in 2024 to 98% in 2026. The three causes of cost overruns are consistent: underestimated token volume as usage scaled, reasoning model hidden token costs that were not in the original models, and agentic workflow costs that multiply token consumption across chains of model calls.

Token price is one variable in the cost equation. Token volume is the other. Most enterprise AI business cases modelled the price. They did not model the volume.

The Three Inference Cost Drivers That Most Enterprises Miss

1. Context length

Every query sent to an AI model includes context. The system prompt that defines how the model should behave. The conversation history. The documents retrieved by the retrieval system. The tool outputs that the agent has collected.

All of this counts as tokens. And context length has been growing rapidly as AI systems become more capable and more connected to enterprise data.

A simple chatbot query might consume 500 tokens. An agent workflow that retrieves multiple documents, maintains conversation history, and executes several tool calls might consume 50,000 tokens for a single completed task. The cost per completed task is 100 times larger than a simple query, even at the same token price.

Managing context length is one of the highest-impact levers in inference cost management. Techniques like prompt compression, context window management, and selective retrieval reduce token consumption per query without reducing output quality.

2. Model selection

Not all queries need the most capable model. A query that asks an AI to classify a customer complaint into one of five categories does not need a frontier model. A query that asks an AI to synthesise a complex legal document and identify regulatory implications does.

Using a frontier model for every query is the fastest way to generate an unsustainable cost structure. Enterprises with mature inference cost management route queries to the smallest model that can complete them reliably. Frontier models handle the tasks that genuinely require their capability. Lightweight models handle the routine tasks that make up the majority of query volume.

This model routing approach is one of the most effective cost reduction strategies available. Enterprises that implement it consistently report 30 to 60% reductions in inference spend with no reduction in output quality on the tasks that matter.

3. Agentic multiplication

AI agents are the fastest-growing source of inference cost in enterprise environments in 2026. As covered in why AI agent costs keep rising, response refinement alone accounts for 60% of agentic AI spend. Each refinement loop is additional inference.

An agent that takes 10 steps to complete a task and refines its output three times at each step generates 30 inference calls for a single completed workflow. At scale, across hundreds of concurrent agent workflows, the multiplication is significant.

Controlling agentic inference costs requires scoping agents tightly, setting token budgets per task, and building circuit breakers that stop runaway refinement loops before they generate unbounded spend.

How to Measure Inference Costs Correctly

Most enterprise AI cost reporting focuses on total spend. This is necessary but not sufficient.

The metric that reveals whether inference costs are under control is cost per completed unit of business value. Not total tokens consumed. Not total spend. Cost per resolved customer query. Cost per completed purchase order. Cost per verified job card closure. Cost per decision generated.

This metric connects inference cost to business outcome. It reveals whether the AI is generating value at a cost that makes the business case work. And it reveals when costs are rising without a corresponding increase in value.

A system whose cost per completed task is stable or declining as it scales is a system whose inference economics are working. A system whose cost per completed task is rising is a system that needs architectural attention before the economics become untenable.

The AI ROI measurement framework applies directly here. Boards do not want to see token spend charts. They want to see cost per resolved ticket, cost per AI-completed workflow, and the relationship between inference spend and business outcomes generated.

What Good Inference Cost Management Looks Like

Enterprises that have brought inference costs under control have typically implemented four practices.

• Visibility before optimisation:

Every inference call is tagged with the workflow, the model, the user or agent that generated it, and the business purpose it served. Without this attribution, cost optimisation is guesswork. With it, the highest-cost workflows are immediately visible and the optimisation priority is clear.

• Model routing by task type:

A defined framework determines which model handles which type of query. Frontier models for complex reasoning and synthesis. Mid-tier models for structured analysis and summarisation. Lightweight models for classification, routing, and simple question answering. The routing logic is automated and applied consistently.

• Token budgets per workflow and agent:

Each workflow and each agent has a defined token budget. When the budget is approached, the system flags it. When it is exceeded, the system either stops or escalates for human review. This prevents runaway spend on individual tasks and makes cost behaviour predictable.

• Caching for repeated queries.

Many enterprise AI workflows involve queries that are similar or identical across users. Semantic caching stores the results of previous queries and serves them for subsequent similar queries without generating a new inference call. For high-volume enterprise applications, caching can reduce inference calls by 20 to 40%.

Who Owns Inference Costs in Your Organisation

This is the question that most enterprises have not yet answered clearly.

AI inference cost sits at the intersection of technology, finance, and the business functions that use AI. The technology team deploys the systems that generate the costs. The finance team manages the budget. The business functions determine how much usage is generated.

Without clear ownership, inference costs drift upward as each function assumes someone else is managing them. With clear ownership, the accountability for cost per business outcome sits with someone who has both the authority to make architectural decisions and the incentive to connect costs to outcomes.

The fastest-growing new job title in enterprise technology in 2026 is some variation of AI Cost Engineer or LLM FinOps Lead. This reflects the recognition that inference cost management is a specialised discipline that requires dedicated ownership.

For enterprises that do not yet have this ownership defined, the who owns AI question has a practical cost dimension that goes beyond governance. It is a cost management question as much as an accountability question.

Vishleshan AI's forward deployed engineering (FDE) approach builds inference cost governance into every AI deployment from the start. Model routing, token budgets, caching architecture, and cost attribution are architectural components of the system, not optimisations added after the bill arrives. Book a Consultation