LiteLLM
Listed at https://litellm.ai
💰 Token Pricing
| Type | Price | Note |
|---|---|---|
| Input | 开源免费自部署；LiteLLM Cloud 按量计费 | per million tokens |
| Output | 开源免费自部署；LiteLLM Cloud 按量计费 | per million tokens |
🤖 Supported Models (100)
✨ Pros
- ✓100+ provider unified interface, popular open-source
- ✓Self-hosted free, data stays on private network
- ✓Provider fallback, load balancing, budget controls
- ✓One-click Docker deployment
- ✓Fast-growing active GitHub community
- ✓Custom provider and model mapping support
⚠️ Cons
- ×Self-hosting requires ops (Docker/server)
- ×No built-in SaaS observability
- ×No encryption/Guardrails out of box
- ×Advanced features need custom dev
- ×Non-standard API provider adapters limited
🎯 Best For
Teams needing self-hosted API gateways; data privacy-sensitive apps; developers avoiding SaaS gateway fees
💰 Pricing & Plans
| Plan | Price | License | Best For |
|---|---|---|---|
| Open Source | $0 | MIT-style | Self-hosted teams wanting zero gateway cost, full control, and data staying on private networks |
| Enterprise | Sales-quoted (annual) | Commercial (self-host or air-gapped) | Organizations needing SSO/RBAC, audit logs, secret management, and support at scale |
🔧 API & Developer Experience
- •OpenAI-Compatible Endpoint: Point any OpenAI SDK at the LiteLLM proxy's `/chat/completions` (plus `/responses`, `/embeddings`, `/images`, `/audio`, `/moderations`, `/batches`) by swapping the base URL — no code rewrite for most apps.
- •100+ Providers in One API: OpenAI, Anthropic, Azure, Gemini, AWS Bedrock, Vertex, vLLM, Ollama, Mistral, Groq, xAI and more — including Chinese providers (Dashscope/Qwen, DeepSeek, ModelScope, Moonshot/Kimi, Volcengine, Xinference).
- •Config-Driven Routing: Model routes, keys, budgets and fallbacks live in a single `config.yaml`; restart to hot-reload routing without redeploying application code.
- •Resilience & Traffic Control: Per-request provider fallbacks/failover, weighted load balancing, and A/B or canary routing across models and vendors keep calls alive during rate limits and degradation.
- •Virtual Keys & Governance: Mint per-app or per-user virtual keys mapped to real upstream keys, with budgets, RPM/TPM rate limits, and spend tracking by key, user, team and org.
- •Caching & Guardrails: Prompt and LLM caching cut cost/latency; built-in guardrails cover PII, jailbreak and prompt-injection before upstream calls; Prometheus metrics and alerting included.
- •Deployment Options: Run via `pip install litellm` or the Docker image (`litellm --model ...`), self-hosted or air-gapped, with an admin dashboard UI and optional LiteLLM Enterprise.
🛠️ Open-Source Unified Gateway
LiteLLM's core strength is being the de-facto open-source layer that makes every LLM look like one consistent OpenAI-style API. Instead of learning a dozen vendor SDKs, a team installs the proxy, declares routes in `config.yaml`, and calls a single local `http://0.0.0.0:4000` endpoint for text, embeddings, image and audio. Because the gateway holds the keys, budgets and fallbacks, policy lives in one auditable place rather than scattered across applications. Crucially it is self-hostable, so teams that cannot send sensitive prompts to a third-party SaaS still get enterprise routing, failover, caching and spend tracking on infrastructure they control. The result is a control plane that travels with any provider strategy, and an active community keeps adapters current as providers ship new models.
🌐 China Access & Latency
Because LiteLLM is a self-hostable Python/Docker application, it runs anywhere a container runs—including mainland China servers and Chinese clouds. A China-based team can deploy the proxy on an Alibaba, Tencent, or Huawei VM and route to domestic providers (Dashscope, DeepSeek, Moonshot/Kimi, Volcengine, ModelScope) over fast in-country links with single-digit millisecond RTT, avoiding any cross-border hop for those models. This makes it a natural unified gateway for China-first products that also want occasional access to overseas frontier models (which still require an exit and a stable outbound path). The hosted LiteLLM Enterprise control plane and SaaS dashboards are operated by a U.S. vendor (Berrie AI) and are not China-resident, so they are typically reached via proxy; for strict data-residency workloads the air-gapped Enterprise self-host is the recommended route.