The organisations spending the most on AI in 2026 are not the ones with the most ambitious strategies. They are the ones that deployed AI into production without understanding how the cost engine actually works.
Deloitte's 2026 State of Generative AI report delivers an uncomfortable finding: inference — the process of running AI models in production — now accounts for 85% of enterprise AI spending. Not hardware acquisition. Not model training. Not integration. The ongoing cost of simply using the AI you already deployed.
For Hong Kong enterprise leaders who approved AI pilots and are now reviewing quarterly spend reports, this guide explains what inference is, why costs escalate rapidly, and the framework for taking back control.
What You Will Know by the End of This Guide
By the end of this article, you will have a working definition of AI inference, understand why production costs behave so differently from pilot costs, and have a three-tier decision framework for managing inference spend without degrading business outcomes.
What Is AI Inference?
AI inference is the process of using a trained AI model to generate a response or prediction. When a user types a query into your AI-powered tool and receives an answer, that is inference happening in real time. Every call to an AI model — whether it is summarising a document, classifying a customer complaint, or generating a report — is an inference event. Inference is distinct from model training, which is the computationally intensive process of teaching a model using large datasets. Training happens once (or periodically). Inference happens every time the model is used.
Why Enterprise Inference Costs Spiral Out of Control
Inference costs feel manageable during pilots because usage is controlled and volumes are low. The economics change dramatically when AI moves into production for three structural reasons.
First, token consumption scales with complexity. Large language models charge by the token — roughly three-quarters of a word. A simple query uses hundreds of tokens. An agentic AI workflow — where the model reasons through multiple steps, accesses tools, and cross-checks outputs — can use tens of thousands of tokens per task. Gartner's 2025 AI Infrastructure report found that agentic AI architectures use 5 to 30 times more tokens per task than single-turn queries. If you built your AI business case on single-turn usage patterns and then deployed agentic workflows, your cost model is structurally wrong.
Second, context windows amplify spend. Modern AI models can process enormous amounts of text in a single session — some up to one million tokens. Organisations that feed AI assistants with full document libraries, complete email threads, or large data exports on every query are consuming context window capacity at rates that make per-query costs jump by orders of magnitude. Every token sent to the model costs money, whether the model needed that information to answer the question or not.
Third, the provider economics are strained. Even the largest AI providers are operating on tight unit economics. Industry analysis suggests that frontier model providers like OpenAI spend approximately US$1.35 in compute costs for every US$1 of revenue generated. That margin pressure means inference pricing is unlikely to fall as fast as enterprise adoption is growing. The assumption that costs will "sort themselves out" as AI scales is not supported by current provider economics.
The Three-Tier Inference Architecture Framework
Enterprise leaders who have brought inference costs under control are not doing so by restricting AI use. They are applying a structured approach to matching model capability to task complexity — what the emerging FinOps for AI discipline calls tiered inference architecture.
Tier 1 — Frontier Models: Reserved for genuinely complex reasoning tasks where accuracy is mission-critical. Legal analysis, risk assessment, complex document drafting, multi-step strategic synthesis. These tasks justify the premium cost of frontier models like GPT-4o, Claude Opus, or Gemini Ultra because the output quality directly affects business decisions. Frontier models should typically handle fewer than 15% of your total inference volume.
Tier 2 — Mid-Range Models: The operational workhorse layer. Suitable for most business processes that require strong language capability but not frontier-level reasoning: customer correspondence, internal report generation, data summarisation, structured extraction from documents. Models in this category — GPT-4o Mini, Claude Sonnet, Gemini Flash — cost 60 to 90% less than frontier equivalents while delivering performance that is indistinguishable for most business tasks.
Tier 3 — Lightweight and Specialised Models: High-volume, low-complexity tasks that are driving most of your inference bill. Classification, routing, sentiment tagging, keyword extraction, simple Q&A against structured data. These tasks can often be handled by fine-tuned smaller models or purpose-built classifiers at a fraction of frontier costs. Several Hong Kong financial services firms have reduced inference spend by 40 to 60% by moving classification and routing workloads from frontier models to specialised tier-3 options.
Model Routing: The Mechanism That Makes Tiering Work
Knowing you should use different models for different tasks is not useful without a mechanism to implement it. Model routing is the architectural pattern that automatically directs each inference request to the appropriate tier based on the characteristics of the query.
A routing layer sits between your application and your AI providers. It evaluates incoming requests against defined criteria — query complexity, required accuracy, latency tolerance, data sensitivity — and routes each request to the optimal model. Simple requests go to tier-3 models at low cost. Complex requests escalate to frontier models. Routine requests are handled mid-tier.
For enterprise leaders evaluating model routing implementations, three questions matter. Does the routing logic account for task-specific accuracy requirements, or does it treat all queries as homogeneous? Can the routing thresholds be adjusted without engineering intervention as your AI use cases evolve? Does the system provide audit trails that map each inference request to its cost and model tier, enabling ongoing optimisation?
Common Mistakes in Enterprise Inference Management
Three patterns consistently drive unnecessary inference cost in enterprise deployments. First, pilot-to-production context window inheritance: organisations that built their pilot prompts with maximum context for quality reasons and then carried those prompts directly into production without optimising for the production scale. A prompt that works well with a 50,000-token context window during testing can drive enormous costs when executed 10,000 times per day.
Second, uniform model deployment: treating all AI tasks as equivalent and routing everything through a single frontier model because it is the easiest configuration. This is the vendor default — and it benefits the vendor, not you.
Third, absent inference observability: deploying AI into production without token-level monitoring. If you cannot see which workflows are consuming the most tokens, you cannot optimise. FinOps for AI — applying the cost allocation and optimisation practices of cloud infrastructure management to AI inference — is emerging as a dedicated function in forward-looking enterprise technology teams. Organisations that establish inference observability before costs become a board-level concern are significantly better positioned to scale AI responsibly.
The Strategic Implication for Hong Kong Enterprise Leaders
The inference cost challenge is not a signal to slow down AI adoption. It is a signal to architect AI investments more deliberately. Organisations that deploy AI into production with a tiered infrastructure model and model routing in place from the start are not just managing costs — they are building the operational foundation that allows AI to scale without the spend escalation that is forcing competitors to retreat from promising initiatives.
The leaders who understand inference economics today will be the ones presenting credible AI scale plans to their boards tomorrow.
Understanding inference economics is step one. Identifying the right architecture for your specific workloads — and the partner who can implement it — is step two. UD's team will walk you through every step: from AI readiness assessment to inference architecture design, vendor selection, and ongoing cost optimisation. 28 years of Hong Kong enterprise IT experience, applied to your AI investment.
