As enterprise AI adoption moves past the pilot stage and into operations, the nature of the traffic that applications generate is changing as well. What used to be a structure where one application called one model and managed one API key has shifted into a structure where multiple model providers, prompt-driven applications, and early-stage agent systems all run at the same time. The problem is that most of this traffic flows straight from application code to the model provider. Without a point in the middle where policy can be applied, security teams have nowhere to inspect traffic, token usage grows unmetered, and every new provider added brings with it another integration with unclear ownership.
Kong has framed this problem from a proxy perspective. Its approach is to route every request between applications and AI providers through a reverse proxy, and to apply security, governance, and cost control at that point. This article summarizes the three AI traffic patterns Kong has identified, the roles a unified proxy needs to perform, and a phased approach to adoption.
☑️ What Is an AI Traffic Proxy
Proxying AI traffic means routing every request between applications and AI providers through a reverse proxy, and applying security, governance, and cost control at that point. An AI gateway is the centralized reverse proxy that performs this role, sitting between applications and every AI destination to apply policy in one place. This allows authentication, rate limiting, cost control, and observability to be managed at the infrastructure layer rather than in application code.
☑️ Why Control Breaks Down with Direct Integration
According to Stanford HAI's 2025 AI Index Report, organizational AI adoption reached 78% in 2024, up from 55% the year before. As the scale of adoption grows, running multiple providers simultaneously has become the norm, and Kong points to three problems that grow together as a result.
Security exposure
When application code calls model providers directly, credentials, prompts, and responses pass through the network without a central inspection point. Every integration becomes an attack surface. Typical examples include API keys left sitting in client-side storage, or sensitive data leaving the network in plaintext, breaking compliance boundaries.
Unpredictable cost
Token consumption is hard to predict and easy to abuse. Gartner projected in a March 2025 announcement that global generative AI spending would reach $644 billion in 2025, and in a September 2025 announcement estimated that total AI spending would reach roughly $1.5 trillion. Both figures are projections as of their announcement dates, but it's clear that spending at this scale has become a board-level risk. Every time the same query triggers a repeated, uncached model call, it consumes budget.
Operational fragility
Each provider has different SDKs, authentication models, and rate-limiting behavior. Bolting governance onto each integration individually makes it difficult to maintain consistency. Without a proxy, adding one more provider means duplicating authentication handling, rewriting rate-limiting logic, and modifying every service that calls the model. Work that should have been a configuration change becomes a code change, and that burden repeats across every team building AI features.
☑️ Three AI Traffic Patterns That Need to Be Controlled
Kong breaks down the routing types that platform teams need to control in an operational AI environment into three categories. Each pattern has a different character in terms of security, cost, and orchestration.
App-to-LLM traffic (North-South)
This is the direct path from an application to a model provider — for example, a chatbot calling an external LLM API. This path requires centralized authentication, application-level rate limiting, logging, and guardrails. Because this path handles the raw data payload directly, the proxy needs to control both the connection and the content. That means infrastructure-level controls (centralized authentication, failover when a provider goes down, rate limiting) need to work together with data-level guardrails (PII sanitization, semantic caching, prompt injection defense).
Agent-to-agent traffic, A2A (East-West)
This is communication between autonomous agents — one agent delegating a task to another — and it can cross service boundaries or even organizational boundaries. Governance in this area requires establishing verifiable agent identity, applying agent-level authorization boundaries, and blocking infinite loops that cascade and drain tokens. Without a control point, agents end up performing actions on behalf of other systems without cryptographic identity verification, and organizations lose visibility into rapidly growing non-human traffic.
MCP traffic
MCP (Model Context Protocol) traffic is the path connecting autonomous agents to local data sources, file systems, and development tools — the destinations an agent calls to complete a task. Proxying this protocol requires protocol-aware authentication, fine-grained authorization at the level of individual tool calls, and management of context-window thresholds, in order to safely limit what a model can see and do. Left uncontrolled, shadow MCP servers spring up across the organization, and agents end up accessing internal infrastructure with no audit trail.
☑️ Five Functions a Unified AI Proxy Performs
Kong lays out five functions that a proxy needs in order to act as the single ingress and egress point for all AI requests. The structure has applications, agents, and tools point at this single location instead of the provider directly, applying policy before a request ever reaches its destination.
First, unified endpoint abstraction: a single stable API sits in front of every provider so applications aren't locked into a specific vendor in code.
Second, multi-provider load balancing: routing and failover across models happen without touching application code.
Third, token-based rate limiting: consumption is capped by token count rather than request count, controlling actual cost.
Fourth, semantic prompt caching: semantically identical prompts return a cached response, reducing both latency and spend.
Fifth, protocol-aware routing for MCP and A2A: agent and tool protocols are handled as their native protocol rather than as generic HTTP.
☑️ How an AI Gateway Differs from an API Gateway
An API gateway protects and routes service-to-service request and response traffic, but it has no concept of tokens, prompts, model providers, or agent protocols. An AI gateway adds an AI-specific layer on top of that: token-based rate limiting, semantic caching, prompt guards, and protocol-aware routing. The core of managing AI traffic is keeping the existing controls platform teams expect — authentication, authorization, observability, rate limiting — while applying AI-specific functions from the same single control plane.
☑️ How Kong AI Gateway Handles the Three Traffic Types
Kong AI Gateway positions itself as a product that controls LLM, MCP, and A2A traffic from a single gateway, mapping each pattern to a function on top of one runtime.
LLM governance
This covers how developers, applications, and agents consume LLMs — from access permissions to data leakage to token usage. Per the product page, this includes blocking data leakage through PII sanitization, improving LLM traffic efficiency through semantic caching, routing, and load balancing, and protecting resources through semantic prompt guards and access control. Because the gateway abstracts provider authentication, credentials never sit in application code. Kong's own materials cite the semantic cache as delivering roughly 3–10x latency improvement, with proportional cost savings.
MCP governance
This covers how MCP servers are created and how agents discover and consume them. The product page specifies automatic generation of MCP tools and servers on top of APIs Kong manages, authentication for controlling access to MCP servers, and token-spend management through MCP context optimization. MCP traffic is subject to call-level authorization.
A2A governance
This applies authentication, observability, and audit tracing to multi-agent traffic. All A2A traffic is observed and A2A-specific metrics are collected, capturing telemetry per call — payload, latency, token usage, and errors. Centralized authentication and authorization are applied across A2A traffic, and every A2A RPC call can be audited and traced, including caller identity and the function being called.
Token Quotas and AI Observability
This layer operates underneath all three patterns. Multi-LLM load balancing applies across OpenAI, Anthropic, Gemini, Mistral, and self-hosted models, with token-based rate limiting to cap spend. On the quota side, quotas for LLM consumption and token spend can be set by user, model, and time period, and enforced at the gateway layer. It's also possible to build showback and chargeback structures for LLM, agent, and MCP usage. On observability, AI consumption, tool usage, and token spend can be tracked, with logging and tracing used to identify points of AI exposure.
☑️ A Phased Approach to Adoption
There's no need to control all three patterns from day one. Kong proposes a phased rollout.
Phase 1 centralizes app-to-LLM traffic: applications are switched to point at the gateway, provider credentials are moved out of code, and application-level logging and rate limiting are turned on. This phase alone closes the largest security gap.
Phase 2 adds cost control: semantic caching and token-based rate limiting are applied to reduce spend on repeated prompts, capping uncontrolled consumption before it shows up on the bill.
Phase 3 extends to agents and MCP: A2A and MCP traffic are brought under the same control plane, with agent identity, tool-call authorization, and audit logging added as those patterns move into production.
Under this structure, switching or adding a provider becomes a configuration change rather than a code change.
☑️ FAQ
What is an AI gateway?
An AI gateway is a centralized reverse proxy that routes and controls traffic between applications and AI providers. It applies authentication, rate limiting, cost control, and observability to LLM calls, prompts, agent traffic, and tool calls, managing AI governance at the infrastructure layer rather than in application code.
How is an AI gateway different from an API gateway?
An API gateway protects and routes service-to-service request/response traffic but has no concept of tokens, prompts, model providers, or agent protocols. An AI gateway adds token-based rate limiting, semantic caching, prompt guards, and protocol-aware routing on top of standard gateway functionality, because a general-purpose API gateway alone can't control AI traffic.
What is an MCP proxy?
An MCP proxy routes traffic between AI agents and MCP servers through a central gateway. The proxy authenticates requests, authorizes individual tool calls, enforces context limits, and logs every interaction, letting platform teams centrally control which tools and data sources agents can access.
How should LLM API calls be rate-limited?
The standard is to limit by token count rather than request count, since cost and load scale with tokens consumed. The gateway measures token usage per application, team, or agent and enforces quotas before a request ever reaches the provider.
How do you manage multiple LLM providers without modifying application code?
Route through an AI gateway that provides a unified endpoint and abstracts away provider-specific authentication and API differences. The application calls a single interface, while the gateway handles routing, failover, and load balancing. Under this structure, adding or switching providers becomes a configuration-level change.
What is agent-to-agent (A2A) routing?
A2A routing controls communication between autonomous agents, including A2A protocol-based traffic. The gateway issues and verifies agent identity, applies agent-level rate limiting, and keeps an audit record of actions an agent performs on behalf of another agent — treating agent communication as operational traffic at the same level as APIs.
☑️ Conclusion
Operational AI environments run on top of three traffic patterns: app-to-LLM, agent-to-agent communication, and MCP. Managing these one integration at a time creates security gaps, cost-control failures, and operational complexity. That's the case for a proxy structure that controls all three patterns from a single control plane.
In environments with strong audit-trail and data-control requirements — finance, public sector, manufacturing — the adoption sequence and scope need to be reviewed together. CloudNetworks can help review a customer's AI traffic structure, determine which pattern to bring under control first, and work out how to connect it with existing API management systems. If you're evaluating Kong AI Gateway, please reach out through the link below.
▶ Learn more about Kong AI Gateway
[Source : Kong, "How to Proxy Every AI Traffic Pattern Through One Gateway", Kong, "Secure, Scalable AI Gateway for AI Connectivity", Stanford HAI, "The 2025 AI Index Report — Economy", Gartner, "Gartner Forecasts Worldwide GenAI Spending to Reach $644 Billion in 2025", Gartner, "Gartner Says Worldwide AI Spending Will Total $1.5 Trillion in 2025"]
