Enterprise AI adoption is moving past the experimentation stage and into production environments. Within organizations, multiple LLMs and AI agents—and more recently MCP (Model Context Protocol) servers—are rapidly proliferating. The problem is that when each application implements its own authentication, cost control, data protection, and logging separately, the same work gets duplicated, and it becomes difficult to get a unified view of an organization's overall AI traffic.
Against this backdrop, there is a growing need for a layer positioned between applications and AI models that connects and controls traffic in one place. This is the role that an AI gateway plays. This article summarizes what Kong AI Gateway is, what problems it solves, and what capabilities it uses to manage LLM, MCP, and agent traffic.
☑️ What Is Kong AI Gateway?
Kong AI Gateway is an AI connectivity and governance layer positioned between applications and LLMs, agents, and MCP servers, handling AI traffic routing, authentication, cost control, security, and observability all within a single gateway. The key point is that it controls not only LLM traffic but also MCP traffic and agent-to-agent (A2A) traffic through the same gateway.
Kong AI Gateway is built on top of Kong API Gateway. This means the policies and plugins used in existing API gateways—authentication/authorization, traffic control, rate limiting, transformation, and more—can be applied to AI traffic as well. Through this gateway, Kong is designed to let developers, applications, and agents manage how they consume AI, covering everything from access permissions to data leakage and token usage.
☑️ Why an AI Gateway Is Necessary
Without an AI gateway, each application connects directly to multiple LLMs, and common functions such as security, retries, logging, caching, and prompt management end up being implemented separately in each application. This results in duplicated development of the same functionality, inconsistent governance levels across teams, and difficulty achieving unified visibility at the organizational level. In addition, the scope that needs to be controlled in an AI environment is broader than with traditional APIs. Customer information and sensitive data must be prevented from flowing into LLMs, consumption across multiple providers and models must be managed under a single standard, and token usage and costs must be continuously monitored.
An AI gateway separates these common concerns from application code and moves them into the gateway layer. By centrally handling functions such as request routing, authentication, authorization, rate limiting, and traffic monitoring, it allows AI to be treated as a single service provided by the organization's platform.
☑️ How Kong AI Gateway Works
Kong AI Gateway acts as a proxy between applications and AI models. Instead of connecting directly to each provider, applications send requests to the gateway address. From a code perspective, integration can be achieved simply by changing the endpoint (base_url) in the LLM call to the gateway address, without exposing individual provider authentication keys directly to the application. Through this low-code integration, Kong supports moving GenAI projects to a production-ready level.
Structurally, Kong Enterprise consists of a Control Plane, which handles management functions, and a Data Plane, which processes actual traffic. AI Gateway and MCP Gateway are components of the Data Plane, operating on the same platform alongside API Gateway, Event Gateway, Ingress Controller, and Service Mesh.
AI requests passing through the gateway go through a staged plugin pipeline. According to introductory materials, this pipeline is composed of the following stages in order: Security & Access Control, Cost Control & Caching, Content Safety, Context Management, LLM Proxying, Response Content Safety, and Logging & Analytics. Only the policies needed at each stage are selected and applied.
☑️ Key Features of Kong AI Gateway
Multi-LLM Support and Provider Switching
Kong AI Gateway allows multiple AI providers to be used together through a unified API interface. Providers can be switched simply by changing configuration, making it possible to apply new use cases or switch to another provider to maintain availability if a specific provider experiences an outage.
In customer environments, this feature reduces dependency on a specific provider and allows model replacement or redundancy configurations to be handled through gateway settings rather than application code changes.
LLM Policies and Security Guardrails
Kong AI Gateway applies multiple layers of guardrails to LLM traffic. A representative example is PII sanitization, which prevents personal information, passwords, code, and similar data from flowing into LLMs, reducing data leakage and compliance risk. According to the version 3.10 release materials, this feature covers more than 20 PII types and 12 languages, and depending on configuration, also supports an option to restore the original data—masked at the response stage—back to the end user.
Semantic Prompt Guard manages requests on a specific topic using deny rules. For example, if political questions are set as a blocked category, the system recognizes and blocks requests that fall under that topic in meaning, even if worded differently. This helps ensure content moderation, protect corporate image, and maintain a safe AI usage environment.
In addition, existing Kong Gateway authentication and access control policies—such as OpenID Connect, ACL (Access Control List), and OPA (Open Policy Agent)—can be applied directly to AI traffic.
Cost Control and Performance Optimization
Kong AI Gateway provides Semantic Caching. Semantic caching returns already-stored responses for prompts that are semantically similar, even if worded differently, reducing repeated LLM calls to cut costs and minimize response latency. It supports various AI embedding models and connection formats for vector databases (e.g., Redis, Postgres, Valkey, and managed services).
Semantic routing and load balancing distribute requests to specialized models depending on prompt type. For example, technical support inquiries can be routed to a relatively higher-performance model, while internal policy or benefits questions can be routed to a lightweight model. This allows the appropriate model to be called for each prompt type and optimizes model usage and cost.
It also allows quotas for LLM consumption and token usage to be set and applied at the gateway level, based on user, model, and time period. This data can also be used to build showback and chargeback systems for LLM, agent, and MCP usage.
AI Traffic Observability
Kong AI Gateway provides L7 observability for AI traffic. AI consumption, tool usage, token consumption, response status codes, and usage and cost by provider and by user can all be viewed on a dashboard. This allows costs to be monitored and usage patterns to be tuned, while logging and tracing can be used to debug AI traffic.
Metrics available on the observability dashboard include usage by model, usage share by model provider, response error rate, and token usage and cost by requester and by user.
☑️ Governance That Extends to Agent and MCP Traffic
Kong AI Gateway's management scope is not limited to LLM traffic. The same gateway also controls MCP server traffic and agent-to-agent (A2A) traffic.
MCP Gateway
MCP provides a standardized way to integrate agents/AI applications with LLMs and tools. However, the MCP standard itself is still under continuous development and change, and challenges such as security issues and organizational deployment guidelines remain, so it is not a complete solution on its own.
Kong MCP Gateway addresses this gap. It can MCP-enable existing REST APIs managed by Kong by mapping them to MCP tools and servers (MCP Proxy plugin), and it supports OAuth 2.1, the standard authorization method for the MCP protocol (MCP OAuth2 plugin). It also allows existing Kong Gateway traffic control and security policies—such as rate limiting, routing, load balancing, IP restriction, and OPA evaluation—to be applied directly to MCP traffic.
Proxy modes are divided by purpose: routing requests to an existing MCP server, performing protocol conversion on a REST API to provide an MCP endpoint, providing only an MCP endpoint, and performing MCP/REST protocol conversion for a REST API. Mapping REST APIs to MCP functionality increases productivity by removing the need to develop MCP separately, and avoids exposing individual authentication tokens and secrets.
Agent Gateway (A2A)
In multi-agent environments where multiple agents collaborate, authentication and control are needed at every connection point between agents. Kong Agent Gateway applies centralized authentication/authorization (AuthN/Z), rate limiting, observability, and structured logging to A2A traffic.
Specifically, it collects detailed telemetry—including payload, latency, token usage, and errors—for A2A calls to identify bottlenecks, prevents unauthorized agent interactions, and maintains audit trail records for all A2A RPC calls, including caller information and the functions and results invoked.
Extending Across the Full AI Data Path
AI traffic is not made up of AI-native traffic alone. Agents also consume context from an organization's APIs and event data. Kong AI Gateway observes and controls the application layer, intelligence layer, and context layer within a single platform, helping to provide enterprise data, events, and APIs in a form that agents can use.
☑️ Frequently Asked Questions
What kind of product is Kong AI Gateway?
Kong AI Gateway is an AI connectivity and governance layer positioned between applications and LLMs, MCP servers, and AI agents, handling AI traffic routing, authentication, security, cost control, and observability within a single gateway. It manages LLM, MCP, and A2A traffic through the same gateway.
Do I need to make major changes to my existing application code?
Integration can be achieved through low-code changes, such as switching the endpoint of the LLM call to the gateway address. Individual provider authentication keys can be managed at the gateway level instead of being exposed directly in the application.
Can I use multiple LLM providers together?
Yes. Multiple AI providers can be used through a unified API interface, and providers can be switched simply by changing configuration. If a specific provider experiences an outage, you can switch to another provider to maintain availability.
Can it manage MCP or agent traffic in addition to LLM traffic?
Kong AI Gateway manages MCP server traffic and agent-to-agent (A2A) traffic through the same gateway. It provides MCP-enablement, OAuth 2.1 authorization, and A2A authentication, observability, and audit logging.
How is AI cost controlled?
Semantic caching reduces repeated calls, semantic routing calls the appropriate model for each prompt type, and token quotas by user, model, and time period are applied at the gateway level. Token usage and cost by provider and by user can be checked on the observability dashboard.
☑️ Conclusion
Kong AI Gateway is an AI connectivity and governance layer designed to connect and control growing LLM, MCP server, and AI agent traffic through a single gateway. By moving authentication, security, cost control, and observability—previously scattered across individual applications—into the gateway, it enables AI to be operated as a single service provided by the organization's platform.
Cloud Networks reviews the adoption of such AI gateways tailored to customer environments and supports their implementation and operation. If you'd like help figuring out where to start—multi-LLM integration, MCP/agent traffic governance, token cost optimization, or AI traffic observability—please reach out using the contact information below.
▶ Learn more about Kong AI Gateway
[Source: Kong, "Secure, Scalable AI Gateway for AI Connectivity," https://konghq.com/products/kong-ai-gateway; Kong, "Announcing Kong's New Open Source AI Gateway," https://konghq.com/blog/product-releases/announcing-kong-ai-gateway; Kong, "Kong AI Gateway Launches Next-Gen Capabilities to Enhance AI Governance, Help Reduce LLM Hallucinations and Provide Infrastructure for Agentic Workflows," https://konghq.com/company/press-room/press-release/kong-ai-gateway-launches-next-gen-capabilities-to-enhance-ai-governance-help-reduce-llm-hallucinations-and-provide-infrastructure-for-agentic-workflows]
