If you’re running more than a few thousand AI API calls per day, you’re almost certainly leaving money on the table. OpenAI’s Batch API and Anthropic’s Message Batches API both offer a flat 50% discount — the catch is that your requests finish within 24 hours instead of in real time. For a surprisingly large share of production workloads, that tradeoff is completely acceptable.
What the Batch API Actually Is (and Isn’t)
Both OpenAI and Anthropic have separate API endpoints designed for asynchronous, high-volume workloads. You submit a file of requests — up to 50,000 individual prompts in OpenAI’s case — and the provider processes them during off-peak hours, returning results within 24 hours. The pricing discount is exactly 50% versus the standard synchronous API rate.
This is not a beta feature or a hidden workaround. OpenAI made its Batch API generally available in 2024, and Anthropic followed with Message Batches shortly after. Both are production-grade, with SLAs, quota limits, and dedicated documentation.
What batch processing is not: it is not a cheaper way to power a chatbot, a real-time translation widget, or any feature where a user is actively waiting. The 24-hour window is a hard constraint, not a soft guideline. If your use case requires a response in under a few seconds, batch is simply the wrong tool.
When Batch Makes Financial Sense
The math is straightforward: if your monthly API spend is $2,000 today and you can shift 60% of requests to batch, you save $600 per month, or $7,200 per year, with zero change to model quality or output format. Before you assume your workloads can’t tolerate async processing, audit what you’re actually calling the API for.
Common workloads that are genuinely asynchronous and batch-ready:
- Content enrichment pipelines: tagging, classifying, or summarizing existing documents nightly
- SEO metadata generation: title, description, and schema markup generated for a product catalog on a schedule
- Sentiment analysis: scoring customer feedback, reviews, or support tickets that don’t need instant scoring
- Lead enrichment: generating company summaries or contact research for CRM records added during the day
- Report generation: producing AI-drafted sections of weekly reports that go out Monday morning
In our experience with NMM students running production AI systems, roughly 40-60% of their API volume can shift to batch without any user-facing impact. That’s a meaningful reduction. To understand the full cost picture before and after, use the free AI Token Counter to measure your actual token consumption per task and estimate batch versus sync cost at your current volume.
OpenAI Batch API: Implementation Walkthrough
The OpenAI Batch API uses .jsonl files — one JSON object per line, each representing a single API request. Here is the minimal structure:
{"custom_id": "req-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize this: <text>"}], "max_tokens": 200}}Step 1: Create your JSONL file. Each line gets a unique custom_id — this is how you match outputs back to inputs. Keep IDs meaningful (e.g., product-sku-1234) rather than sequential integers.
Step 2: Upload the file. Use the /v1/files endpoint with purpose: "batch". The API returns a file_id.
Step 3: Submit the batch. POST to /v1/batches with your file_id, endpoint: "/v1/chat/completions", and completion_window: "24h". You receive a batch_id immediately.
Step 4: Poll for completion. GET /v1/batches/{batch_id} to check status. When status is "completed", the response includes an output_file_id.
Step 5: Download results. GET /v1/files/{output_file_id}/content to retrieve the output JSONL. Each line maps back to your custom_id.
The full round-trip for a 10,000-request batch typically completes in 2-6 hours in practice, well within the 24-hour window. Build your pipeline to check status every 30 minutes rather than polling aggressively.
Anthropic Message Batches: Key Differences
Anthropic’s implementation is conceptually identical but has a few structural differences worth noting. Batches are submitted as a JSON array (not a .jsonl file), each item containing a custom_id and a params object that mirrors the standard /v1/messages request body. The endpoint is /v1/messages/batches.
Anthropic’s pricing follows the same 50% discount principle. As of mid-2026, Claude 3.5 Haiku via batch costs $0.40 per million input tokens versus $0.80 synchronously. Claude 3.5 Sonnet drops from $3.00 to $1.50 per million input tokens in batch mode. At scale, those numbers add up fast.
One practical difference: Anthropic’s batch window is also 24 hours, but results are streamed as a Server-Sent Events stream when you download them, not a single file download. Your retrieval code needs to handle this, but any SSE client library makes it trivial.
Both APIs support the same models available on their synchronous endpoints, so you are not giving up model capability — only response latency.
Error Handling and Quotas You Should Know
Batch jobs are not immune to errors. Individual requests within a batch can fail (due to content policy, malformed input, or context length violations) without failing the entire batch. The output JSONL includes an error field for failed rows — always process errors separately from successes.
OpenAI’s default batch quota is 100,000 queued tokens per model per minute, and total in-flight batch size is capped at 200,000 requests or 50M tokens across all pending batches. If you exceed these, the batch submission will fail. Check your account’s batch quota under “Rate limits” in the OpenAI dashboard and request increases if you’re hitting ceilings.
Anthropic imposes per-account limits on concurrent batch jobs. For most accounts this is 100 requests per batch call and 10 concurrent batches. Enterprise accounts get higher limits on request.
Modeling the True Savings Before You Migrate
Before refactoring your codebase, run the numbers. Token costs vary by model, and the batch discount applies uniformly, but you should also account for:
- Engineering time: refactoring synchronous pipelines to async takes real hours
- Infrastructure changes: you need a job queue, a status checker, and result storage
- Edge cases: what happens when a batch job fails? You need a fallback path
A rough framework: if your monthly AI spend in a workflow is above $500 and the latency shift is acceptable, the engineering investment (typically 4-8 hours for a well-documented pipeline) pays back within 2-3 months. Below $200/month, the ROI is marginal unless you already have an async job system in place.
Use the AI Token Counter to get a precise monthly token estimate for each workflow before you commit to the migration. Input your average prompt length, expected call volume, and target model — the tool outputs both sync and batch cost estimates side-by-side so you can size the opportunity accurately.
Get Your Batch Cost Estimate in 30 Seconds
Stop estimating on a spreadsheet. Paste a sample prompt into the AI Token Counter, enter your monthly call volume, and select your model. The tool shows you current sync pricing, effective batch pricing at 50% off, and annual savings — all without signing up for anything.
Frequently asked questions
Does the Batch API use the same model quality as the synchronous API? Yes. Batch requests run on the same model weights as real-time requests. The only difference is scheduling: your requests are queued and processed during periods of lower demand. Output quality, context length limits, and feature support (like function calling and JSON mode) are identical.
What happens if my batch job doesn’t complete within 24 hours? OpenAI and Anthropic both guarantee the 24-hour completion window as part of the API contract. In practice, most batches complete in 2-8 hours. If a batch does exceed 24 hours — which is rare and typically caused by service-side issues — you can cancel and resubmit. Neither provider charges for incomplete or cancelled batches.
Can I mix different models in a single batch file? With OpenAI, each batch job targets a single endpoint and model — you specify the model per request in the body, so technically you can mix GPT-4o and GPT-4o-mini within one batch file. Anthropic requires you to specify the model per request as well. The billing and quota accounting, however, is per-model, so verify your limits apply to each model separately.
Is there a minimum batch size to get the discount? No minimum. A batch with a single request still qualifies for 50% off. In practice, submitting individual requests as single-item batches adds unnecessary latency and operational complexity — the discount only makes practical sense when you have at least dozens of requests to group together.
How do I handle partial failures in a large batch?
Build your retrieval script to separate successful rows from error rows on download. For each failed custom_id, log the error code and requeue just those requests in a follow-up batch or via the synchronous API. Never resubmit the entire batch — you’ll double-bill the requests that already succeeded.
Related reading
- AI Token Counter — measure your token usage and estimate batch vs sync costs
- Small Language Models and Cost Savings
- AI Cost Projection and Budgeting Framework
