As enterprises move generative AI into full-scale operation, the center of gravity for costs is shifting rapidly. In the past, cloud infrastructure such as servers, storage, and networks was at the core of cost management, but now tokens used for LLM calls and GPUs used for AI inference are emerging as new cost categories.
In a recent blog post, Cast AI co-founder Laurent Gil described a customer spending six to seven times its cloud costs on GPU inference, explaining that tokens are becoming the new cloud cost. Based on that blog post and materials on Cast AI's LLM optimization and GPU optimization solutions, this article outlines why token costs have become a new focus and how they can be approached from an operational perspective. It is intended as a reference for cloud, infrastructure, FinOps, and AI leaders at organizations that are considering AI adoption or already operating LLM-based services.
☑️ What It Means That Tokens Are the New Cloud Cost
The statement that tokens are the new cloud cost means that LLM token consumption and GPU inference costs incurred in operating AI services have become core cost items that must be managed as much as, or even more than, traditional cloud infrastructure costs. In AI-centric organizations, AI consumption is already becoming the largest cost area. In other words, token costs are not a one-time expense but an operating cost that is incurred continuously and fluctuates with usage. For this reason, the approach used to manage cloud costs, namely gaining visibility, applying governance, and continuous optimization, must be applied to tokens as well.
☑️ Why Token Costs Have Become a New Focus
As organizations move AI from experimentation to real service operations, AI usage itself is growing rapidly. At FinOps X 2026, the trend of sharply rising GPU inference spending was clearly evident.
The evolution of cost management can be traced as follows. DevOps introduced the concept of financial accountability to software development, and FinOps applied the same accountability to cloud infrastructure. Here, FinOps refers to an operating practice that applies financial accountability and management frameworks to cloud usage costs. Its core is gaining cost visibility, applying governance, and continuously optimizing usage and costs. Now, the same financial management applied to AI consumption, known as Tokenomics, is emerging as a new discipline. The launch of the Tokenomics Foundation was announced at FinOps X 2026, and Laurent Gil serves as a founding board member of the foundation.
From an operational perspective, token costs are difficult to control because they are directly tied to developer and user activity. As usage increases, costs rise with it, and limiting usage to cut costs can reduce productivity and output quality. This dilemma is the key factor that makes token cost management difficult.
☑️ Not All Tokens Are Equal
A common mistake in managing token costs is treating all AI spending the same way. Not all tokens are the same. LLMs differ not only in quality tier but also in specialization by use case, so the model suited to complex coding tasks is different from the model suited to simple queries. Choosing the wrong model can affect not only cost but also output quality.
Cast AI also offers a practical benchmark. Only about 5% of developers' day-to-day tasks actually require a top-performing frontier model, while the remaining 95% can be handled by simpler, cheaper, or open-source models with no difference in results. If the same top-tier model is used for every task, the organization ends up paying frontier model prices even for routine work.
☑️ The Key to Token Efficiency Is Automatic Model Routing
So how can the right model be chosen for each task? The direction is clear. Rather than leaving model selection to individual developers, the infrastructure layer makes the decision automatically. Cast AI describes this as using AI to choose AI.
In this structure, developers define only the outcome they want, and the system selects and runs the appropriate model for each part of the task. Developers only need to check the results without knowing which model handled which step. This follows the same principle as how Cast AI optimizes Kubernetes infrastructure. Just as autonomous agents handle placement, instance selection, and costs without users having to think about instance types, the selection of the right model is automated in the token domain as well.
From an operational perspective, this difference is significant. When people judge model selection every time, consistency is hard to maintain, and they tend to fall back on familiar high-end models. By contrast, automating routing makes it possible to handle routine work, which accounts for 95% of tasks, with cost-efficient models and to use frontier models only for tasks that truly need them.
☑️ Token Limits Can Be the Wrong Guardrail
Setting hard caps on token usage to reduce costs has clear limitations. It can be compared to handing developers a laptop with a one-hour battery that can only be charged once a day. When tokens run out mid-task, the work stops.
Once budgets are set and token limits are configured, developers may hit a wall at the most critical moment, just before solving a problem. Blocking users is closer to friction disguised as control than to governance.
The alternative is to give developers sufficient access while optimization works quietly in the background. This structure uses open-source models where appropriate, applies automatic model routing, and handles cost management without developers having to think about it. The approach is for FinOps teams and infrastructure to absorb the complexity so that users can focus on results. For this operating model to work financially, autonomous automation, in which the system selects, recovers, and optimizes on its own, must be in place.
☑️ How Cast AI Approaches Token Costs
Cast AI aims to be a platform that goes beyond showing costs to automatically optimizing and taking action. The Cast AI Router in its LLM optimization solution is the capability that actually implements the automatic model routing described above, and the GPU optimization solution manages the GPU infrastructure costs incurred when running self-hosted models.
LLM Optimization: AI Enabler and Cast AI Router
AI Enabler is a solution that lowers application costs by routing requests to the most suitable and cost-efficient LLM. It is where the automatic model routing discussed above is implemented as a concrete product capability. Its key features are as follows.
The AI Enabler proxy integrates with a wide range of LLM providers, including OpenAI, Anthropic, Mistral, and Databricks, and supports both streaming and non-streaming responses.
GPU Optimization: OMNI Compute for AI
When running self-hosted models, not only token pricing but also the cost of the GPU infrastructure that runs those models becomes something to manage. OMNI Compute for AI enables scarce GPU and compute capacity distributed across multiple clouds and regions to be operated within a single Kubernetes cluster, helping AI teams scale without refactoring applications or increasing operational burden. Its key features are as follows.
☑️ Frequently Asked Questions
Q. What does it mean that tokens are the new cloud cost?
A. It means that LLM token consumption and GPU inference costs incurred in operating AI services have become as important to manage as traditional cloud infrastructure costs. In AI-centric organizations, AI consumption is already becoming the largest cost area.
Q. Why not use the highest-performing model for every task?
A. Using the top-tier model uniformly for every task means paying frontier model prices even for routine work. According to Cast AI, only about 5% of tasks actually require a frontier model, and the rest can be handled by cheaper or open-source models.
Q. Do developers have to choose the right model for each task themselves?
A. Rather than leaving model selection to developers, having the infrastructure route requests automatically is recommended. Cast AI's Cast AI Router automatically routes requests to the optimal LLM based on performance, cost, and provider limits, and integrates using the standard OpenAI API format.
Q. Won't setting a cap on token usage reduce costs?
A. Hard caps can reduce costs, but developers may run out of tokens mid-task, lowering productivity. Instead of blocking usage, the suggested alternative is to provide sufficient access while optimizing automatically in the background.
Q. Which LLMs does AI Enabler support?
A. The AI Enabler proxy integrates with a wide range of LLM providers, including OpenAI, Anthropic, Mistral, and Databricks, and supports both streaming and non-streaming responses.
☑️ Conclusion
As tokens and GPU inference establish themselves as new cost categories, the time has come to apply the principles used to manage cloud costs, namely visibility, governance, and continuous optimization, to AI as well. Cast AI presents an operating model in which LLM token costs are optimized through automatic model routing and even GPU capacity for self-hosted models is managed automatically, so that cost optimization works in the background while users focus on results.
In an environment where token and GPU costs are rising rapidly, moving from checking costs after the fact to an operating framework that automatically optimizes and takes action is worth considering. When evaluating adoption, organizations can review whether they have sufficient visibility into current LLM usage costs and GPU utilization, whether model routing suited to task characteristics is in place, and whether they have a structure that can automatically respond to GPU capacity shortages and cost fluctuations.
As an official partner of Cast AI, CloudNetworks supports the adoption, implementation, and operation of AI cost optimization tailored to each customer's environment and regulatory requirements. If managing token costs and GPU utilization is a concern, please contact us through the link below.
▶ Cast AI Learn More
