APIRank
EN 中文

LiteLLM 2026: The Open-Source AI Gateway That Routes 100+ LLMs Through One API

What is LiteLLM, and why does it matter in 2026?

LiteLLM is the de facto open-source standard for routing requests across multiple LLM providers. Founded in 2023 by Krrish Dholakia and Ishaan Jaffer as a thin Python wrapper around the OpenAI SDK, it has evolved into a 53,599-star GitHub project (verified July 2026) that today powers AI access at Netflix, Lemonade, Rocket Money, and hundreds of platform teams shipping LLM-backed products. The company is Y Combinator-backed and currently transitioning its proxy server from Python to a Rust core for performance.

The core promise is simple: write one completion() call against the OpenAI schema, and LiteLLM will route it to OpenAI, Anthropic, Google Vertex, AWS Bedrock, Azure, Cohere, Hugging Face, NVIDIA NIM, Ollama, vLLM, or any of 100+ other providers. Behind that single interface sits a self-hostable AI Gateway (Proxy) with virtual keys, per-team budgets, rate limiting, fallback chains, load balancing, prompt caching, guardrails, and full OpenTelemetry-compatible observability. As of July 2026 the project has served 1 billion+ requests and pulled 240 million+ Docker images.

LiteLLM pricing in 2026: free OSS, paid Enterprise

LiteLLM is unusual in the AI infrastructure space: the entire core proxy is MIT-licensed open-source software that you can self-host with no per-request fees. The only costs are the underlying GPU or API spend from the providers you route to. For teams that need commercial features (SSO, JWT auth, audit logs, custom SLAs, dedicated support, air-gapped deployment), LiteLLM offers an Enterprise tier with pricing on application — public price cards do not exist, but the Netflix and Lemonade case studies suggest Enterprise contracts in the $50K-$500K/year range depending on request volume.

Compared to Portkey (which starts at $49/month for the Hobby plan and $199/month for Growth) and Cloudflare AI Gateway (which adds $0.20 per million Gateway requests on top of provider fees), LiteLLM's open-source-first model is the most cost-effective path for platform teams that already have Kubernetes/ops capacity. The trade-off is operational ownership: you run the proxy, monitor it, scale it, and patch it.

How the LiteLLM Proxy works (with code)

The simplest way to use LiteLLM is via the Python SDK. The completion() function accepts a model string in the format provider/model, and LiteLLM handles authentication, retries, and provider-specific quirks:

from litellm import completion
import os

os.environ['OPENAI_API_KEY'] = 'sk-...'
os.environ['ANTHROPIC_API_KEY'] = 'sk-ant-...'

# Call any of 100+ providers with the same interface
response = completion(
    model='anthropic/claude-opus-4-8',
    messages=[{'role': 'user', 'content': 'Explain transformers in 3 sentences.'}],
    fallback_models=['openai/gpt-5', 'google/gemini-2.5-pro'],
)
print(response.choices[0].message.content)

For team-scale usage, the AI Gateway (Proxy) runs as a Docker container or Kubernetes deployment. A minimal config.yaml looks like:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: local-llama
    litellm_params:
      model: ollama/llama3.3
      api_base: http://ollama.internal:11434

router_settings:
  routing_strategy: least-busy
  num_retries: 3
  timeout: 30

litellm_settings:
  drop_params: true
  set_verbose: false

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

# Launch
# $ litellm --config config.yaml --port 4000

Once running, the proxy exposes an OpenAI-compatible /v1/chat/completions endpoint that any client (the OpenAI SDK, Anthropic SDK with a shim, Vercel AI SDK, LlamaIndex, LangChain) can talk to. The admin UI is at /ui and provides virtual key management, spend tracking, and team budgets.

Cost tracking, budgets, and rate limiting

LiteLLM's strongest feature for platform teams is its cost attribution layer. Every request is tagged with a virtual key (mapped to a user, team, or org) and the response includes an accurate cost calculation based on the model's published token rates. Costs can be logged to s3, GCS, Postgres, or any OTEL-compatible backend. Teams can set hard budgets at the key level — when a team exceeds its budget, requests return 429 with a structured error.

Rate limiting is similarly granular. The proxy tracks TPM (tokens per minute) and RPM (requests per minute) per model and per deployment, and supports cooldown periods when a deployment hits rate limits. This is critical for production deployments that talk to OpenAI or Anthropic under tight rate ceilings — LiteLLM will automatically failover to a backup deployment rather than 500ing.

Fallback, load balancing, and traffic mirroring

LiteLLM's Router supports three patterns that map directly to production scenarios:

Fallback chains: Define primary + backup models. If the primary 429s or 5xxs, the router tries the next model. The 'try in order, fail fast' pattern is great for cost-optimization (cheap model first, premium only when needed) and for resilience (OpenAI + Anthropic + Bedrock as backup).

Load balancing: Spread requests across N deployments of the same model. Useful when one provider has rate limits that the others don't. Strategies include simple-shuffle, least-busy, usage-based, and latency-based.

Traffic mirroring: Copy a percentage of production traffic to a secondary deployment for evaluation. The canonical use case is A/B testing a new model against the incumbent — both get the same prompt, but only the primary's response goes back to the user, while the secondary's response is logged for offline eval.

LiteLLM vs Portkey vs Cloudflare AI Gateway vs OpenRouter

These four tools overlap heavily but target different buyers:

Tool Pricing model Open source Best for
LiteLLM OSS free + Enterprise (custom) Yes (MIT) Platform teams that self-host; cost control + routing
Portkey $49/mo Hobby, $199/mo Growth, Enterprise No (SaaS) Teams needing observability + guardrails without ops overhead
Cloudflare AI Gateway $0.20/M requests (1M free) No (SaaS) Existing Cloudflare users; simple caching + analytics
OpenRouter Per-token markup on 300+ models No (SaaS) Exploration / access to many models without commit

If you are a platform team with Kubernetes/ops capacity and need fine-grained control over routing, cost attribution, and self-hosted observability, LiteLLM is the strongest choice. If you want a SaaS-only product with minimal setup, Portkey or Cloudflare AI Gateway is faster. If you want access to many models without committing to a single provider, OpenRouter is the right call.

MCP server, guardrails, and what's new in 2026

Three new features in 2026 make LiteLLM materially stronger:

MCP server (Q1 2026): LiteLLM now exposes any deployed model as a Model Context Protocol tool to Claude Desktop, Cursor, Cline, and any MCP-compatible agent runtime. This means a single config.yaml turns your proxy into an MCP tool registry — engineers building agents can call Claude, GPT, and local Llama models from the same place.

Guardrails (Q1 2026): Built-in guardrail support for PII detection, jailbreak protection, content moderation, and custom regex checks. Guardrails can be configured per-model or per-key, and failures can be configured to either block (return 400) or log-and-pass (with the violation logged for review).

Rust core (Q2 2026, in progress): The proxy server is being rewritten in Rust with PyO3 bindings for the Python SDK compatibility layer. Early benchmarks show 5-10x throughput improvement on the hot path. Existing Python configs continue to work unchanged.

When NOT to use LiteLLM

Despite its strengths, LiteLLM is not the right fit for every team. Skip LiteLLM if:

You need a SaaS console: LiteLLM's UI is functional but not as polished as Portkey or Helicone. If your non-technical stakeholders need a dashboard, plan to build a thin internal UI on top of the proxy's API.

You are routing to a single provider: If 100% of your traffic goes to OpenAI or Anthropic, the proxy adds latency and operational overhead without much benefit. Use the provider's SDK directly.

You cannot operate a Kubernetes cluster: LiteLLM is meant to be self-hosted. There is no hosted LiteLLM-as-a-Service option (only the Enterprise tier, which is self-hosted with support). If you have no ops capacity, choose a SaaS tool.

Frequently asked questions

What is LiteLLM used for?

LiteLLM is used to route requests across 100+ LLM providers (OpenAI, Anthropic, Google Vertex, AWS Bedrock, Azure, Cohere, Hugging Face, NVIDIA NIM, Ollama, vLLM) using a single OpenAI-compatible interface. It is the standard tool for platform teams that need to give developers access to many models while maintaining cost attribution, rate limiting, and observability. Netflix, Lemonade, and Rocket Money use LiteLLM in production.

How much does LiteLLM cost?

LiteLLM's open-source proxy is MIT-licensed and free to self-host — you only pay the underlying provider costs (OpenAI tokens, Anthropic tokens, GPU spend for self-hosted models). The Enterprise tier adds SSO, JWT auth, audit logs, custom SLAs, dedicated support, and air-gapped deployment; pricing is on application. Public price cards do not exist, but case studies suggest Enterprise contracts in the $50K-$500K/year range depending on request volume.

Is there a LiteLLM free tier?

Yes — the open-source proxy is fully free with no rate limits and no request caps. You can serve unlimited requests through a self-hosted LiteLLM Proxy. The only costs are the upstream provider API spend or the GPU cost of self-hosted models. No credit card is required to start.

Can I use LiteLLM from inside China?

Yes, via self-hosting. LiteLLM's open-source proxy runs anywhere Docker or Kubernetes runs — including on Aliyun ECS, Tencent Cloud, or Huawei Cloud inside China. The proxy talks to the upstream providers (OpenAI, Anthropic, etc.) using your own keys, so if you need to access the providers from inside China, you'll need a stable proxy or your providers need to be available locally (e.g. DeepSeek, Qwen, GLM via Aliyun Bailian). The official LiteLLM SaaS does not currently serve China-based customers.

Does LiteLLM support OpenAI-compatible API calls?

Yes. LiteLLM's primary interface is the OpenAI format — the proxy exposes /v1/chat/completions, /v1/embeddings, /v1/models, and other OpenAI-compatible endpoints. The OpenAI SDK, Anthropic SDK (with a thin shim), Vercel AI SDK, LangChain, LlamaIndex, and any tool that speaks OpenAI format will work with LiteLLM by changing the base URL.

How does LiteLLM compare to Portkey?

Portkey is a SaaS AI gateway with built-in observability, guardrails, and prompt management — faster setup, prettier UI, but no open-source option and higher per-month cost (Hobby $49/month, Growth $199/month). LiteLLM is open-source and self-hosted with similar routing and cost-tracking capabilities but requires you to operate the infrastructure. For a team that values data control and avoids SaaS lock-in, LiteLLM is the better choice. For a team that wants zero-ops with a polished UI, Portkey wins.

How does LiteLLM compare to OpenRouter?

OpenRouter is a multi-provider model marketplace with 300+ models accessed through a single API key and pay-per-token pricing — zero ops, instant setup, but with a 5-30% markup on provider costs and no self-hosted option. LiteLLM is open-source and routes to the same providers, but you bring your own provider keys, you pay provider-direct prices, and you operate the proxy yourself. OpenRouter is faster to start; LiteLLM is cheaper at scale and gives full control.

How does LiteLLM compare to Cloudflare AI Gateway?

Cloudflare AI Gateway is a SaaS request router with caching, analytics, and fallback — best for existing Cloudflare users who want lightweight routing without ops overhead. LiteLLM goes much deeper on cost attribution, per-team budgets, MCP integration, guardrails, and traffic mirroring, but you have to operate it. For a team that lives inside Cloudflare Workers, AI Gateway is the natural fit. For a team that needs platform-grade control, LiteLLM wins.

What is the LiteLLM MCP server?

The LiteLLM MCP server (released Q1 2026) exposes every model in your proxy as a Model Context Protocol tool to Claude Desktop, Cursor, Cline, and any MCP-compatible agent runtime. A single config.yaml turns your LiteLLM proxy into a multi-model MCP tool registry — agents can call Claude, GPT, and local Llama models through the same tool interface.

Does LiteLLM have an affiliate program?

LiteLLM does not currently have a public affiliate program. For teams that need an affiliate-aligned CTA, an alternative is FreeModel, an OpenAI-compatible aggregator that routes across multiple providers and adds cost-routing optimization for self-deployed models.