### The Cost Anatomy of an LLM Call Every API call has five cost components. ```mermaid graph LR A[User Query] --> B[System Prompt
500-2000 tokens] A --> C[Retrieved Context
500-4000 tokens] A --> D[User Message
50-500 tokens] B --> E[Input Cost
$2.50/1M tokens] C --> E D --> E E --> F[Model Processing] F --> G[Output Cost
$10.00/1M tokens] ``` System prompts are the silent killer. A 1,500-token system prompt sent with every request costs $3.75 per million requests just for that prefix. At 100K requests per day, that is $375/day -- $11,250/month -- for text that never changes. ### Provider Caching: Built-in Discounts All three major providers offer provider-side prompt caching in 2026, but the mechanics differ. See Phase 11 · 15 for the deep dive. | Provider | Mechanism | Discount | Minimum | Cache Duration | |----------|-----------|----------|---------|----------------| | Anthropic | Explicit cache_control markers | 90% on cache hits (pay 25% extra on write) | 1,024 tokens (Sonnet/Opus), 2,048 (Haiku) | 5 min default; 1h extended (2x write premium) | | OpenAI | Automatic prefix matching | 50% on cache hits | 1,024 tokens | Best-effort up to 1 hour | | Google Gemini | Explicit CachedContent API | ~75% reduction (plus storage) | 4,096 (Flash) / 32,768 (Pro) | User-configurable TTL | **Anthropic's approach** is explicit. You mark sections of your prompt with `cache_control: {"type": "ephemeral"}`. The first request pays a 25% write premium. Subsequent requests with the same prefix get a 90% discount. A 2,000-token system prompt that costs $0.005 normally costs $0.000625 on cache hits. Over 100K requests, that saves $437.50/day. **OpenAI's approach** is automatic. Any prompt prefix that matches a previous request gets a 50% discount. No markers needed. The tradeoff: less discount, less control, but zero implementation effort. ### Semantic Caching: Your Custom Layer Provider caching only works for identical prefixes. Semantic caching handles the harder case: different queries with the same meaning. "What is the return policy?" and "How do I return an item?" are different strings but identical intent. A semantic cache embeds both queries, computes cosine similarity, and returns the cached response if similarity exceeds a threshold (typically 0.92-0.95). ```mermaid flowchart TD A[User Query] --> B[Embed Query] B --> C{Similar query
in cache?} C -->|sim > 0.95| D[Return Cached Response] C -->|sim < 0.95| E[Call LLM API] E --> F[Cache Response
with Embedding] F --> G[Return Response] D --> G ``` The embedding costs are negligible. OpenAI's text-embedding-3-small costs $0.02 per million tokens. Checking the cache costs almost nothing compared to a full LLM call. ### Exact Caching: Hash and Match For deterministic calls (temperature=0, same model, same prompt), exact caching is simpler and faster. Hash the full prompt, check the cache, return if found. This works perfectly for: - System prompt + fixed context + identical user queries - Function calling with identical tool definitions - Batch processing where the same document gets processed multiple times ### Rate Limiting: Protecting Your Budget Rate limiting is not just about fairness. It is about survival. **Token bucket algorithm:** each user gets a bucket of N tokens that refills at rate R per second. A request consumes tokens from the bucket. If the bucket is empty, the request is rejected. This allows bursts (use the full bucket at once) while enforcing an average rate. **Per-user quotas:** set daily/monthly token limits per user tier. | Tier | Daily Token Limit | Max Requests/min | Model Access | |------|------------------|------------------|-------------| | Free | 50,000 | 10 | GPT-4o-mini only | | Pro | 500,000 | 60 | GPT-4o, Claude Sonnet | | Enterprise | 5,000,000 | 300 | All models | ### Model Routing: Right Model for the Right Job Not every query needs GPT-4o. "What time does the store close?" does not require a $10/M-output model. GPT-4o-mini at $0.60/M output handles it perfectly. Claude Haiku at $1.25/M output handles it. A simple classifier routes cheap queries to cheap models and complex queries to expensive models. ```mermaid flowchart TD A[User Query] --> B[Complexity Classifier] B -->|Simple: lookup, FAQ| C[GPT-4o-mini
$0.15/$0.60 per 1M] B -->|Medium: analysis, summary| D[Claude Sonnet
$3.00/$15.00 per 1M] B -->|Complex: reasoning, code| E[GPT-4o / Claude Opus
$2.50/$10.00+] ``` A well-tuned router saves 40-70% on model costs alone. ### Cost Tracking: Know Where the Money Goes You cannot optimize what you do not measure. Log every API call with: - Timestamp - Model name - Input tokens - Output tokens - Latency (ms) - Computed cost ($) - User ID - Cache hit/miss - Request category This data reveals which features are expensive, which users are heavy consumers, and where caching has the most impact. ### Batching: Bulk Discounts OpenAI's Batch API processes requests asynchronously at a 50% discount. You submit a batch of up to 50,000 requests, and results come back within 24 hours. Use batching for: - Nightly document processing - Bulk classification - Evaluation runs - Data enrichment pipelines Not for: real-time user-facing queries (latency matters). ### Budget Alerts and Circuit Breakers A circuit breaker stops spending when you hit a limit. Without one, a bug or abuse can burn through your monthly budget in hours. Set three thresholds: 1. **Warning** (70% of budget): send an alert 2. **Throttle** (85% of budget): switch to cheaper models only 3. **Stop** (95% of budget): reject new requests, return cached responses only ### The Optimization Stack Apply these techniques in order. Each layer compounds on the previous ones. | Layer | Technique | Typical Savings | Implementation Effort | |-------|-----------|----------------|----------------------| | 1 | Provider prompt caching | 30-50% | Low (add cache markers) | | 2 | Exact caching | 10-20% | Low (hash + dict) | | 3 | Semantic caching | 15-30% | Medium (embeddings + similarity) | | 4 | Model routing | 40-70% | Medium (classifier) | | 5 | Rate limiting | Budget protection | Low (token bucket) | | 6 | Prompt compression | 10-30% | Medium (rewrite prompts) | | 7 | Batching | 50% on eligible | Low (batch API) | A RAG app applying layers 1-5 typically reduces costs from $22,500/month to $4,000-6,000/month. That is the difference between burning runway and building a business. ### Real Savings: Before and After Here is a real breakdown for a RAG chatbot serving 10,000 DAU. | Metric | Before Optimization | After Optimization | Savings | |--------|--------------------|--------------------|---------| | Monthly LLM cost | $22,500 | $5,200 | 77% | | Avg cost per query | $0.0075 | $0.0017 | 77% | | Cache hit rate | 0% | 52% | -- | | Queries routed to mini | 0% | 65% | -- | | P95 latency | 2,800ms | 900ms (cache hits: 50ms) | 68% | | Monthly embedding cost | $0 | $180 | (new cost) | | Total monthly cost | $22,500 | $5,380 | 76% | The embedding cost for semantic caching ($180/month) pays for itself within the first hour of cache hits.