Groq

💰 Pricing & Plans

🔧 API & Developer Experience

• •OpenAI-Compatible Endpoint: Base URL https://api.groq.com/openai/v1 mirrors OpenAI's /chat/completions and /responses schemas — existing OpenAI SDKs (Python, JS) work after a base-URL and key swap, so migrations require zero code refactor.
• •Native Groq SDK: Official Python and JavaScript/TypeScript SDKs (groq-sdk) provide typed access to chat, speech, vision, batch, and tool-use endpoints, plus streaming and structured outputs out of the box.
• •Prompt Caching: Automatic prefix caching on GPT-OSS 20B/120B and Safeguard 20B cuts cached input tokens 50% with no code changes; volatile storage only (2-hour TTL), and cached tokens don't count toward rate limits.
• •Service Tiers: Four tiers via the service_tier parameter: performance (enterprise-only, lowest latency), on_demand (default), flex (higher throughput, best-effort), and auto (Groq picks the best tier per request).
• •Built-In Tools: Server-side web search (Visit Website), code execution, Wolfram Alpha, and browser search (GPT-OSS models) — no separate agent plumbing required; tool calls are billed at standard rates.
• •Compound Agentic Models: groq/compound and groq/compound-mini orchestrate multi-step workflows with built-in web search, code execution, and browser automation — useful for research/agent workloads that previously needed custom scaffolding.
• •Batch Processing: Asynchronous batch API delivers up to 50% off all tokens; submit up to 24h ahead for bulk evaluation, embedding, or nightly ingestion. Note: prompt-caching discount doesn't stack with batch discount.
• •Audio / Speech Suite: Whisper Large V3 ($0.111/audio-hour) and V3 Turbo ($0.04/audio-hour) plus Orpheus TTS (per-1M-character pricing) round out a full speech pipeline on the same API key.

⚡ LPU Inference Engine & Latency

Groq's signature capability is its deterministic low-latency inference: the original Language Processing Unit (LPU) and the newer LPX architecture (paired with NVIDIA next-gen GPUs) deliver sub-second time-to-first-token and stable token-per-second throughput on every supported model. GPT-OSS 20B reaches 1,000 tokens/sec on GroqCloud, GPT-OSS 120B sustains 500 t/s, and the Whisper transcription tier pushes audio at hundreds of times real-time — figures that GPU-only clouds rarely match for comparable model sizes. The four service tiers (performance / on_demand / flex / auto) expose this latency profile as a tunable: performance gives enterprise customers strict SLAs, while flex trades latency for higher throughput on non-urgent workloads. Combined with prompt caching (50% off cached input tokens on GPT-OSS), batch discounts, and the built-in agentic Compound models, Groq positions itself as the inference layer of choice for real-time chat, voice agents, and high-volume evaluation pipelines rather than a general-purpose model lab.

🌐 China Access & Latency

api.groq.com is not directly reachable from mainland China — the API is hosted in Groq's US data centers and the GFW blocks traffic to console.groq.com and api.groq.com alike. Mainland developers typically access Groq via VPN, an overseas VPS, or a third-party reseller that re-sells Groq access (some aggregator platforms offer GPT-OSS 120B and Qwen 3.6 routes with Chinese-language billing). For non-China regions, latency to GroqCloud is excellent — US east/west clients see 100–300 ms first-token time and sustained 500+ t/s on the OSS tier. The pricing itself is fine for overseas workloads, but the lack of a self-serve China-direct route, no regional PoPs, and the recent move of Llama 3.1/3.3 to enterprise-only (Contact Sales) make Groq a high-performance foreign API rather than a domestic alternative — teams shipping to mainland users should plan for an aggregator or proxy in their architecture.