August 1, 2026•12 min read•Evergreen Team
AI Agent Cost Optimization & Token Management 2026: Complete Guide to Reducing API Spend by 70%
Master AI agent cost optimization techniques, achieve intelligent token management, caching strategies, model selection, and batch processing to significantly reduce API spend.
The Truth About Runaway AI API Costs
2026 surveys show that 83% of enterprises encounter cost overrun issues when using AI APIs. A medium-sized AI application can generate thousands to tens of thousands of dollars in API fees monthly. Main reasons include: repetitive requests, excessive context lengths, inappropriate model selection, and lack of caching strategies.
The good news: by implementing systematic cost optimization strategies, enterprises can reduce API spend by 50-70% while maintaining performance. This article shares proven cost optimization techniques and best practices.
Understanding Token Billing Models
Most AI APIs bill by token. Understanding token composition is the first step in cost control:
- Input Tokens - Prompts, system messages, context
- Output Tokens - Model-generated responses
- Price Differences - Output tokens typically cost 2-4x more than input tokens
- Model Differences - GPT-4 costs 10-30x more than GPT-3.5
# Token cost calculator
class TokenCostCalculator:
def __init__(self):
# 2026 prices (per million tokens)
self.pricing = {
'gpt-4-turbo': {'input': 10.0, 'output': 30.0},
'gpt-4': {'input': 30.0, 'output': 60.0},
'gpt-3.5-turbo': {'input': 0.5, 'output': 1.5},
'claude-3-opus': {'input': 15.0, 'output': 75.0},
'claude-3-sonnet': {'input': 3.0, 'output': 15.0},
'claude-3-haiku': {'input': 0.25, 'output': 1.25},
}
def estimate_cost(self, model: str, input_tokens: int, output_tokens: int) -> float:
"""Estimate API call cost"""
if model not in self.pricing:
raise ValueError(f"Unknown model: {model}")
pricing = self.pricing[model]
input_cost = (input_tokens / 1_000_000) * pricing['input']
output_cost = (output_tokens / 1_000_000) * pricing['output']
return input_cost + output_cost
def compare_models(self, input_tokens: int, output_tokens: int):
"""Compare costs across different models"""
print(f"\nCost comparison for {input_tokens} input + {output_tokens} output tokens:")
print("-" * 60)
for model, pricing in self.pricing.items():
cost = self.estimate_cost(model, input_tokens, output_tokens)
print(f"{model} " + "$" + f"{cost:.4f}")
# Usage example
calculator = TokenCostCalculator()
# Typical scenario: 1000 input tokens, 500 output tokens
calculator.compare_models(1000, 500)
# Output:
# Cost comparison for 1000 input + 500 output tokens:
# ------------------------------------------------------------
# gpt-4-turbo $0.0250
# gpt-4 $0.0600
# gpt-3.5-turbo $0.0013
# claude-3-opus $0.0525
# claude-3-sonnet $0.0105
# claude-3-haiku $0.0009Strategy 1: Intelligent Caching
Exact Match Caching
The simplest caching strategy is to cache completely identical requests. This is very effective for repetitive questions:
import hashlib
import json
from typing import Optional, Dict, Any
import redis
class ExactCache:
def __init__(self, redis_url: str = "redis://localhost:6379"):
self.redis = redis.from_url(redis_url)
self.ttl = 3600 # 1 hour expiration
def _generate_key(self, messages: list, model: str) -> str:
"""Generate cache key"""
content = json.dumps({
'messages': messages,
'model': model
}, sort_keys=True)
return f"ai_cache:{hashlib.sha256(content.encode()).hexdigest()}"
def get(self, messages: list, model: str) -> Optional[Dict[str, Any]]:
"""Get cached response"""
key = self._generate_key(messages, model)
cached = self.redis.get(key)
if cached:
return json.loads(cached)
return None
def set(self, messages: list, model: str, response: Dict[str, Any]):
"""Cache response"""
key = self._generate_key(messages, model)
self.redis.setex(key, self.ttl, json.dumps(response))
# Usage example
cache = ExactCache()
def call_ai_with_cache(client, messages: list, model: str):
"""AI call with caching"""
# Try to get from cache
cached_response = cache.get(messages, model)
if cached_response:
print("✅ Cache hit!")
return cached_response
# Cache miss, call API
print("❌ Cache miss, calling API...")
response = client.chat.completions.create(
model=model,
messages=messages
)
response_data = {
'content': response.choices[0].message.content,
'usage': {
'prompt_tokens': response.usage.prompt_tokens,
'completion_tokens': response.usage.completion_tokens
}
}
# Cache result
cache.set(messages, model, response_data)
return response_data
# Test
import openai
client = openai.OpenAI()
messages = [{"role": "user", "content": "What is the capital of France?"}]
# First call (cache miss)
result1 = call_ai_with_cache(client, messages, "gpt-3.5-turbo")
# Second call (cache hit)
result2 = call_ai_with_cache(client, messages, "gpt-3.5-turbo")Semantic Caching
Semantic caching caches based on question semantic similarity, hitting cache even with different wording:
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import openai
class SemanticCache:
def __init__(self, similarity_threshold: float = 0.95):
self.similarity_threshold = similarity_threshold
self.cache = [] # [(embedding, response, messages)]
self.embed_client = openai.OpenAI()
def _get_embedding(self, text: str) -> np.ndarray:
"""Get text embedding vector"""
response = self.embed_client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return np.array(response.data[0].embedding)
def _extract_query(self, messages: list) -> str:
"""Extract query text from messages"""
for msg in reversed(messages):
if msg['role'] == 'user':
return msg['content']
return ""
def get(self, messages: list, model: str) -> Optional[Dict[str, Any]]:
"""Semantic cache lookup"""
if not self.cache:
return None
query = self._extract_query(messages)
query_embedding = self._get_embedding(query)
# Calculate similarity with all cache items
similarities = []
for cached_embedding, cached_response, cached_messages in self.cache:
sim = cosine_similarity([query_embedding], [cached_embedding])[0][0]
similarities.append(sim)
# Find most similar
max_idx = np.argmax(similarities)
max_sim = similarities[max_idx]
if max_sim >= self.similarity_threshold:
print(f"✅ Semantic cache hit! Similarity: {max_sim:.3f}")
return self.cache[max_idx][1]
return None
def set(self, messages: list, model: str, response: Dict[str, Any]):
"""Add to semantic cache"""
query = self._extract_query(messages)
embedding = self._get_embedding(query)
self.cache.append((embedding, response, messages))
# Limit cache size
if len(self.cache) > 1000:
self.cache.pop(0)
# Usage example
semantic_cache = SemanticCache(similarity_threshold=0.92)
def call_ai_with_semantic_cache(client, messages: list, model: str):
"""AI call with semantic caching"""
# Try semantic cache
cached_response = semantic_cache.get(messages, model)
if cached_response:
return cached_response
# Call API
response = client.chat.completions.create(
model=model,
messages=messages
)
response_data = {
'content': response.choices[0].message.content,
'usage': {
'prompt_tokens': response.usage.prompt_tokens,
'completion_tokens': response.usage.completion_tokens
}
}
# Add to semantic cache
semantic_cache.set(messages, model, response_data)
return response_data
# Test
messages1 = [{"role": "user", "content": "What is the capital of France?"}]
messages2 = [{"role": "user", "content": "Can you tell me the capital city of France?"}]
result1 = call_ai_with_semantic_cache(client, messages1, "gpt-3.5-turbo")
result2 = call_ai_with_semantic_cache(client, messages2, "gpt-3.5-turbo") # Should hit cacheStrategy 2: Model Routing
Automatically select the most appropriate model based on task complexity, avoiding "using a sledgehammer to crack a nut":
import openai
from typing import List, Dict
class ModelRouter:
def __init__(self):
self.client = openai.OpenAI()
# Model hierarchy
self.models = {
'simple': 'gpt-3.5-turbo', # Simple tasks
'medium': 'gpt-4-turbo', # Medium complexity
'complex': 'gpt-4' # Complex tasks
}
def classify_task_complexity(self, messages: List[Dict]) -> str:
"""Classify task complexity"""
# Extract user messages
user_messages = [m['content'] for m in messages if m['role'] == 'user']
combined_text = ' '.join(user_messages)
# Simple heuristic rules
complexity_indicators = {
'complex': ['analyze', 'compare', 'evaluate', 'design', 'architect', 'optimize'],
'medium': ['explain', 'summarize', 'convert', 'translate', 'review'],
'simple': ['what is', 'how to', 'define', 'list', 'show']
}
text_lower = combined_text.lower()
for level, indicators in complexity_indicators.items():
if any(indicator in text_lower for indicator in indicators):
return level
# Length-based judgment
if len(combined_text) > 500:
return 'medium'
elif len(combined_text) > 1000:
return 'complex'
return 'simple'
def route(self, messages: List[Dict]) -> str:
"""Route to appropriate model"""
complexity = self.classify_task_complexity(messages)
model = self.models[complexity]
print(f"Task complexity: {complexity}, routing to: {model}")
return model
def smart_call(self, messages: List[Dict]) -> Dict:
"""Smart call, automatically select model"""
model = self.route(messages)
response = self.client.chat.completions.create(
model=model,
messages=messages
)
return {
'model_used': model,
'content': response.choices[0].message.content,
'usage': {
'prompt_tokens': response.usage.prompt_tokens,
'completion_tokens': response.usage.completion_tokens,
'total_tokens': response.usage.total_tokens
}
}
# Usage example
router = ModelRouter()
# Simple question
simple_messages = [{"role": "user", "content": "What is Python?"}]
result1 = router.smart_call(simple_messages)
print(f"Used model: {result1['model_used']}") # gpt-3.5-turbo
# Complex question
complex_messages = [{"role": "user", "content": "Analyze and compare the architectural trade-offs between microservices and monolithic approaches for a high-traffic e-commerce platform."}]
result2 = router.smart_call(complex_messages)
print(f"Used model: {result2['model_used']}") # gpt-4Strategy 3: Context Compression
Long context is a major cost source. Compressing context can significantly reduce token usage:
import openai
from typing import List, Dict
class ContextCompressor:
def __init__(self):
self.client = openai.OpenAI()
self.max_context_tokens = 2000
def estimate_tokens(self, text: str) -> int:
"""Estimate token count (rough estimate)"""
# English: 1 token ≈ 4 characters
# Chinese: 1 token ≈ 1-2 characters
return len(text) // 3
def compress_context(self, messages: List[Dict], target_tokens: int = 2000) -> List[Dict]:
"""Compress context to target token count"""
# Preserve system messages and latest messages
system_msgs = [m for m in messages if m['role'] == 'system']
user_msgs = [m for m in messages if m['role'] == 'user']
assistant_msgs = [m for m in messages if m['role'] == 'assistant']
# Calculate current token count
total_tokens = sum(self.estimate_tokens(m['content']) for m in messages)
if total_tokens <= target_tokens:
return messages # No compression needed
print(f"Compressing context from {total_tokens} to {target_tokens} tokens...")
# If too many history messages, summarize
if len(assistant_msgs) > 3:
# Keep last 3 conversation turns
recent_msgs = assistant_msgs[-3:] + user_msgs[-3:]
# Summarize earlier conversation
early_msgs = assistant_msgs[:-3] + user_msgs[:-3]
if early_msgs:
summary = self._summarize_conversation(early_msgs)
# Rebuild message list
compressed = system_msgs + [
{'role': 'system', 'content': f"Previous conversation summary: {summary}"}
] + recent_msgs + [user_msgs[-1]]
return compressed
return messages
def _summarize_conversation(self, messages: List[Dict]) -> str:
"""Summarize conversation history"""
conversation_text = "\n".join([
f"{m['role'].upper()}: {m['content']}"
for m in messages
])
response = self.client.chat.completions.create(
model="gpt-3.5-turbo", # Use cheap model for summarization
messages=[
{"role": "system", "content": "Summarize the following conversation in 2-3 sentences, focusing on key points and decisions."},
{"role": "user", "content": conversation_text}
],
max_tokens=150
)
return response.choices[0].message.content
# Usage example
compressor = ContextCompressor()
# Long conversation
long_conversation = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is machine learning?"},
{"role": "assistant", "content": "Machine learning is a subset of AI..."},
{"role": "user", "content": "Can you explain supervised learning?"},
{"role": "assistant", "content": "Supervised learning is..."},
# ... more messages
]
# Compress context
compressed = compressor.compress_context(long_conversation, target_tokens=1000)
print(f"Original messages: {len(long_conversation)}")
print(f"Compressed messages: {len(compressed)}")Strategy 4: Batch API Calls
For non-real-time tasks, using batch APIs can save 50% on costs:
import openai
import json
from typing import List, Dict
import time
class BatchProcessor:
def __init__(self):
self.client = openai.OpenAI()
def create_batch_file(self, requests: List[Dict], output_file: str = "batch_requests.jsonl"):
"""Create batch request file"""
with open(output_file, 'w') as f:
for i, request in enumerate(requests):
batch_request = {
"custom_id": f"request-{i}",
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": request.get('model', 'gpt-3.5-turbo'),
"messages": request['messages'],
"max_tokens": request.get('max_tokens', 500)
}
}
f.write(json.dumps(batch_request) + "\n")
print(f"Created batch file with {len(requests)} requests")
return output_file
def submit_batch(self, input_file: str) -> str:
"""Submit batch job"""
# Upload file
with open(input_file, 'rb') as f:
file_response = self.client.files.create(
file=f,
purpose="batch"
)
# Create batch job
batch_response = self.client.batches.create(
input_file_id=file_response.id,
endpoint="/v1/chat/completions",
completion_window="24h"
)
print(f"Batch submitted: {batch_response.id}")
print(f"Status: {batch_response.status}")
return batch_response.id
def check_batch_status(self, batch_id: str) -> Dict:
"""Check batch job status"""
batch = self.client.batches.retrieve(batch_id)
print(f"Batch {batch_id}:")
print(f" Status: {batch.status}")
print(f" Request counts: {batch.request_counts}")
if batch.status == "completed":
print(f" Output file: {batch.output_file_id}")
print(f" Cost: 50% discount applied!")
return {
'status': batch.status,
'request_counts': batch.request_counts,
'output_file_id': batch.output_file_id if batch.status == "completed" else None
}
def download_results(self, output_file_id: str) -> List[Dict]:
"""Download batch results"""
file_response = self.client.files.content(output_file_id)
results = []
for line in file_response.text.strip().split("\n"):
result = json.loads(line)
results.append(result)
print(f"Downloaded {len(results)} results")
return results
# Usage example
processor = BatchProcessor()
# Prepare batch requests
requests = [
{
'messages': [{'role': 'user', 'content': f'Analyze this text: {i}'}],
'model': 'gpt-3.5-turbo',
'max_tokens': 200
}
for i in range(100)
]
# Create batch file
batch_file = processor.create_batch_file(requests)
# Submit batch job
batch_id = processor.submit_batch(batch_file)
# Wait for completion (in production, should check asynchronously)
time.sleep(60)
# Check status
status = processor.check_batch_status(batch_id)
if status['status'] == 'completed':
results = processor.download_results(status['output_file_id'])
print(f"Processed {len(results)} requests at 50% cost!")Strategy 5: Prompt Optimization
Optimizing prompts can reduce token usage while maintaining output quality:
class PromptOptimizer:
def __init__(self):
self.client = openai.OpenAI()
def optimize_prompt(self, original_prompt: str) -> str:
"""Optimize prompt to reduce token usage"""
optimization_prompt = f"""Optimize this prompt to be more concise while maintaining clarity and effectiveness. Remove redundant words, use abbreviations where appropriate, and keep the core intent.
Original prompt:
{original_prompt}
Optimized prompt (be 30-50% shorter):"""
response = self.client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": optimization_prompt}],
max_tokens=500
)
optimized = response.choices[0].message.content.strip()
original_tokens = len(original_prompt) // 3
optimized_tokens = len(optimized) // 3
savings = original_tokens - optimized_tokens
print(f"Original: {original_tokens} tokens")
print(f"Optimized: {optimized_tokens} tokens")
savings_pct = savings/original_tokens*100
print(f"Savings: {savings} tokens ({savings_pct:.1f}%)")
return optimized
def use_few_shot_efficiently(self, task: str, examples: List[Dict]) -> List[Dict]:
"""Efficient few-shot learning usage"""
# Only select 2-3 most relevant examples
relevant_examples = examples[:3]
messages = [
{"role": "system", "content": f"You are a helpful assistant for {task}."}
]
# Add examples
for example in relevant_examples:
messages.append({"role": "user", "content": example['input']})
messages.append({"role": "assistant", "content": example['output']})
return messages
def implement_prompt_caching(self, system_prompt: str) -> str:
"""Implement prompt caching (OpenAI feature)"""
# For long system prompts, use cached prefix
# This allows subsequent requests to reuse cached tokens
cached_prompt = f"""[CACHED PREFIX]
{system_prompt}
[END CACHED PREFIX]
Now answer the user's question:"""
return cached_prompt
# Usage example
optimizer = PromptOptimizer()
# Optimize long prompt
long_prompt = """You are an expert software engineer with 20 years of experience in multiple programming languages including Python, JavaScript, Java, C++, and Go. You have deep knowledge of software architecture, design patterns, best practices, and modern development methodologies. When answering questions, please provide clear, concise, and accurate information. Include code examples when relevant. Explain complex concepts in simple terms. Consider edge cases and potential issues. Follow industry best practices and conventions."""
optimized = optimizer.optimize_prompt(long_prompt)
# Output:
# Original: 73 tokens
# Optimized: 45 tokens
# Savings: 28 tokens (38.4%)Comprehensive Cost Optimization Dashboard
import pandas as pd
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
class CostOptimizationDashboard:
def __init__(self):
self.usage_data = []
def log_usage(self, model: str, input_tokens: int, output_tokens: int,
cost: float, cached: bool = False, batch: bool = False):
"""Log API usage"""
self.usage_data.append({
'timestamp': datetime.now(),
'model': model,
'input_tokens': input_tokens,
'output_tokens': output_tokens,
'total_tokens': input_tokens + output_tokens,
'cost': cost,
'cached': cached,
'batch': batch
})
def generate_report(self, days: int = 7) -> Dict:
"""Generate cost optimization report"""
df = pd.DataFrame(self.usage_data)
if df.empty:
return {"error": "No usage data available"}
# Filter time range
cutoff = datetime.now() - timedelta(days=days)
df = df[df['timestamp'] >= cutoff]
# Basic statistics
total_cost = df['cost'].sum()
total_tokens = df['total_tokens'].sum()
total_requests = len(df)
# Optimization metrics
cache_hits = df['cached'].sum()
cache_hit_rate = cache_hits / len(df) if len(df) > 0 else 0
batch_requests = df['batch'].sum()
batch_savings = df[df['batch']]['cost'].sum() * 0.5 # 50% discount
# Model distribution
model_costs = df.groupby('model')['cost'].sum().to_dict()
# Daily cost trend
df['date'] = df['timestamp'].dt.date
daily_costs = df.groupby('date')['cost'].sum().to_dict()
report = {
'period_days': days,
'total_cost': total_cost,
'total_tokens': total_tokens,
'total_requests': total_requests,
'avg_cost_per_request': total_cost / total_requests if total_requests > 0 else 0,
'cache_hit_rate': cache_hit_rate,
'cache_hits': cache_hits,
'batch_requests': batch_requests,
'batch_savings': batch_savings,
'estimated_savings': (cache_hits * df['cost'].mean()) + batch_savings,
'model_costs': model_costs,
'daily_costs': daily_costs,
'recommendations': self._generate_recommendations(df)
}
return report
def _generate_recommendations(self, df: pd.DataFrame) -> List[str]:
"""Generate optimization recommendations"""
recommendations = []
# Check cache hit rate
cache_rate = df['cached'].mean()
if cache_rate < 0.3:
recommendations.append("Low cache hit rate ({:.1%}). Consider implementing semantic caching.".format(cache_rate))
# Check model usage
expensive_models = df[df['model'].str.contains('gpt-4|opus', case=False)]
if len(expensive_models) > len(df) * 0.5:
recommendations.append("High usage of expensive models. Consider model routing to optimize costs.")
# Check batch processing opportunities
non_realtime = df[df['model'].str.contains('gpt-3.5', case=False)]
if len(non_realtime) > 100:
recommendations.append(f"{len(non_realtime)} requests could use batch API for 50% savings.")
# Check token efficiency
avg_tokens = df['total_tokens'].mean()
if avg_tokens > 2000:
recommendations.append("High average token usage. Consider context compression.")
return recommendations
def visualize(self, report: Dict):
"""Visualize report"""
fig, axes = plt.subplots(2, 2, figsize=(15, 10))
# Daily cost trend
ax1 = axes[0, 0]
dates = list(report['daily_costs'].keys())
costs = list(report['daily_costs'].values())
ax1.plot(dates, costs, marker='o')
ax1.set_title('Daily Cost Trend')
ax1.set_ylabel('Cost ($)')
ax1.tick_params(axis='x', rotation=45)
# Model cost distribution
ax2 = axes[0, 1]
models = list(report['model_costs'].keys())
model_costs = list(report['model_costs'].values())
ax2.bar(models, model_costs)
ax2.set_title('Cost by Model')
ax2.set_ylabel('Cost ($)')
ax2.tick_params(axis='x', rotation=45)
# Cache hit rate
ax3 = axes[1, 0]
cache_rates = [report['cache_hit_rate'], 1 - report['cache_hit_rate']]
ax3.pie(cache_rates, labels=['Cache Hits', 'Cache Misses'], autopct='%1.1f%%')
ax3.set_title('Cache Performance')
# Optimization metrics
ax4 = axes[1, 1]
metrics = ['Total Cost', 'Batch Savings', 'Cache Savings']
values = [
report['total_cost'],
report['batch_savings'],
report['cache_hits'] * report['avg_cost_per_request']
]
ax4.bar(metrics, values)
ax4.set_title('Cost Breakdown')
ax4.set_ylabel('Cost ($)')
plt.tight_layout()
plt.savefig('cost_optimization_report.png', dpi=150, bbox_inches='tight')
print("Report saved to cost_optimization_report.png")
# Usage example
dashboard = CostOptimizationDashboard()
# Simulate some usage data
import random
for i in range(100):
model = random.choice(['gpt-3.5-turbo', 'gpt-4-turbo', 'gpt-4'])
input_tokens = random.randint(100, 2000)
output_tokens = random.randint(50, 500)
cached = random.random() < 0.4 # 40% cache hit
batch = random.random() < 0.2 # 20% batch processing
# Calculate cost
pricing = {
'gpt-3.5-turbo': (0.5, 1.5),
'gpt-4-turbo': (10.0, 30.0),
'gpt-4': (30.0, 60.0)
}
input_price, output_price = pricing[model]
cost = (input_tokens / 1_000_000 * input_price +
output_tokens / 1_000_000 * output_price)
if batch:
cost *= 0.5 # Batch discount
dashboard.log_usage(model, input_tokens, output_tokens, cost, cached, batch)
# Generate report
report = dashboard.generate_report(days=7)
print("\n" + "="*60)
print("COST OPTIMIZATION REPORT")
print("="*60)
print(f"Period: Last {report['period_days']} days")
print("Total Cost: " + "$" + f"{report['total_cost']:.2f}")
print(f"Total Requests: {report['total_requests']}")
print("Avg Cost/Request: " + "$" + f"{report['avg_cost_per_request']:.4f}")
print(f"\nOptimization Metrics:")
print(f" Cache Hit Rate: {report['cache_hit_rate']:.1%}")
print(f" Batch Requests: {report['batch_requests']}")
print(" Estimated Savings: " + "$" + f"{report['estimated_savings']:.2f}")
print(f"\nRecommendations:")
for rec in report['recommendations']:
print(f" • {rec}")
# Visualize
dashboard.visualize(report)Best Practices Checklist
- Implement multi-layer caching strategy (exact cache + semantic cache)
- Use model routing, select models based on task complexity
- Optimize prompt length, remove redundant content
- Use batch APIs for non-real-time tasks (save 50%)
- Implement context compression, limit history message count
- Monitor token usage, set cost alerts
- Regularly review and optimize caching strategies
- Use cost dashboard to track optimization effects
Related Tools
If you need cost analysis and monitoring tools, check out our JSON Formatter, CSV to JSON, and Base64 Encoder/Decoder.
Frequently Asked Questions
Why are AI API costs so high?
AI API costs are primarily determined by token usage. Both input and output tokens are billed, and long contexts, repetitive requests, and inefficient prompts can cause costs to rise rapidly.
How to reduce token usage?
Methods to reduce token usage include: 1) Use caching to avoid repetitive requests, 2) Optimize prompt length, 3) Choose appropriately sized models, 4) Use batch APIs, 5) Implement context compression.
What is semantic caching?
Semantic caching is a technique that caches AI responses based on semantic similarity rather than exact matching. Even if questions are worded differently, they can hit the cache as long as semantics are similar.
How to choose the right AI model to control costs?
Choose models based on task complexity: use smaller models (like GPT-3.5-turbo) for simple tasks, larger models (like GPT-4) for complex tasks. Implement model routing to automatically select the most cost-effective model.
How much cost can batch APIs save?
Batch APIs typically offer 50% discounts. For non-real-time tasks (like data analysis, content generation), using batch APIs can significantly reduce costs while maintaining the same quality.