Most developers and marketers who use AI APIs have a vague sense that “tokens are like words” — and that works fine until you submit a 40-page legal brief as context, watch your API bill spike, and realize you had no idea how many tokens that was. The relationship between human-readable text and AI tokens is consistent enough to plan around once you know the actual ratios.
How Tokenization Actually Works
Tokenizers don’t split text at word boundaries. They use byte-pair encoding (BPE) or similar algorithms to split text into the most frequent sub-word units found in their training corpus. The result is that common English words are usually one token, uncommon or long words split into two or three tokens, and punctuation and spaces are often bundled with adjacent characters.
OpenAI’s cl100k_base tokenizer (used by GPT-4o, GPT-4, and GPT-3.5-turbo) treats ” the” (with a leading space) as a single token, while “tokenization” splits into ” token” + “ization” — two tokens. Anthropic’s tokenizer for Claude follows similar BPE patterns but is not identical to OpenAI’s, so the same text can produce slightly different token counts on different providers.
The practical implication: you cannot divide your word count by a fixed number and get an exact token count. But you can get close enough for planning — and for exact counts, you should use a dedicated tool.
Token-to-Word Ratios for English Text
For standard English prose — think blog posts, business emails, documentation, or news articles — the ratio is consistently close to 0.75 words per token, or equivalently, 1 token per 0.75 words. That means:
- 1 word ≈ 1.3 tokens
- 100 words ≈ 133 tokens
- 500 words ≈ 667 tokens
For a standard page of text in a document (roughly 250-300 words for a double-spaced academic page, or 400-500 words for a densely typeset business page), token counts work out as:
| Page Type | Approx Words | Approx Tokens |
|---|---|---|
| Double-spaced academic page | 250 | 333 |
| Single-spaced business doc | 450 | 600 |
| Dense typeset (paperback novel) | 500 | 667 |
| Average web article page | 350 | 467 |
A standard 80,000-word novel is approximately 106,000 tokens — which fits comfortably within Claude’s 200K context window, but would require chunking for models with 32K or 128K limits.
To get exact counts for your specific text rather than estimates, paste it directly into the AI Token Counter. The tool uses the actual tokenizer for whichever model you’re targeting and shows token counts with per-model cost breakdowns.
Tokens in Other Languages
Token counts vary significantly across languages, and non-English text almost always costs more tokens per word. This is because BPE tokenizers are trained predominantly on English text, making common English sub-words more efficient (one token per unit) while other languages require more tokens to represent the same semantic content.
Spanish: Spanish is phonetically regular and shares a lot of vocabulary with English (both Latin-derived). The ratio is approximately 1.1-1.2 tokens per word — about 10-20% more expensive than English for the same semantic content. A 400-word Spanish paragraph is roughly 450-480 tokens.
French and Italian: Similar to Spanish. Expect a 10-20% token overhead versus equivalent English text.
German: German compound nouns are long and often split into multiple tokens. German text typically runs 1.2-1.4 tokens per word. Technical German documentation (compounded nouns like “Softwareentwicklungsumgebung”) can push this higher.
Chinese (Simplified/Traditional): This is where the difference becomes significant. Chinese characters don’t have spaces between words, and each character or character pair often becomes a token. The relationship to “words” is less meaningful, but as a rough benchmark, a Chinese character averages about 0.6-0.8 tokens using cl100k_base, while the equivalent English concept might require fewer characters overall. A short Chinese sentence of 20 characters might be 15-20 tokens, while the English equivalent at 10 words is 13-14 tokens. The difference narrows as content gets longer.
Japanese: Similar to Chinese in token inefficiency for BPE tokenizers. Japanese kanji tokenizes roughly 1 token per character, while hiragana strings can token-merge slightly.
Russian and Cyrillic script: Characters outside ASCII are represented in UTF-8 as multi-byte sequences, and tokenizers often produce 2-3 tokens per word for Cyrillic text. Russian text routinely runs 1.5-2x the token count of equivalent English content.
Tokens in Code
Code is distinct from natural language and has its own tokenization patterns. Well-formatted Python, JavaScript, and SQL tend to tokenize more efficiently than natural language because keywords like def, return, SELECT, and WHERE are common enough to become single tokens.
Rough benchmarks for common code types:
| Code Type | Tokens per 100 Characters |
|---|---|
| Python (clean, commented) | 25-35 |
| JavaScript/TypeScript | 28-38 |
| SQL queries | 20-30 |
| JSON data | 30-45 |
| HTML markup | 35-50 |
| CSS | 28-40 |
JSON is particularly expensive because of the repetitive structure: every "key": "value" pattern tokenizes the quotes, colon, and surrounding whitespace separately. A 1,000-character JSON payload might be 300-450 tokens, significantly more than 1,000 characters of Python.
When building RAG pipelines or code generation tools, the JSON overhead matters. If you’re passing large JSON payloads as context, consider serializing to a more compact format or stripping whitespace before sending.
Books, Long Documents, and Context Windows
Understanding tokens per page helps with context window planning. Here are reference points for common long documents:
- Short blog post (800 words): ~1,067 tokens
- Long-form article (3,000 words): ~4,000 tokens
- 10-page report (5,000 words): ~6,667 tokens
- 40-page legal brief (20,000 words): ~26,667 tokens
- Novel (80,000 words): ~106,667 tokens
- Full academic thesis (100,000 words): ~133,333 tokens
GPT-4o’s 128K context window fits roughly 96,000 words or a ~320-page paperback novel. Claude’s 200K window fits approximately 150,000 words — a full-length nonfiction book plus your system prompt and instructions. Gemini 1.5 Pro’s 1M token window fits approximately 750,000 words — multiple books at once.
For enterprise document processing, these numbers define your chunking strategy. A 100-page contract at ~50,000 tokens will not fit in GPT-4o-mini’s 128K context unless your system prompt is minimal.
Why Knowing Your Counts Saves Money
Token counts directly determine your API bill. A 10% reduction in prompt length across a high-volume pipeline translates to a 10% reduction in input token costs. For a pipeline spending $2,000/month, that’s $200/month — saved by tightening prompts, removing redundant instructions, or switching from verbose JSON context to compact text.
Before you optimize, measure. Use the AI Token Counter to get exact token counts for your prompts and context windows, then model the cost at your actual call volume. The tool also shows per-model pricing so you can see whether switching from GPT-4o to GPT-4o-mini (at 1/20th the cost) makes sense for your specific token load.
Count Your Tokens Before You Budget
Stop estimating. Paste your actual prompt into the AI Token Counter — it shows exact token counts using the real tokenizer for your target model, plus the resulting API cost at current prices. Takes 20 seconds and removes all the guesswork from your AI budget.
Frequently asked questions
Are tokens the same across GPT-4o, GPT-4o-mini, and Claude? No. Different model families use different tokenizers. GPT-4o and GPT-4o-mini share the cl100k_base tokenizer, so token counts are identical between them — pricing differs but counts don’t. Claude uses Anthropic’s own tokenizer, which produces slightly different counts for the same text. The difference is usually small (under 5%) for English text but can be larger for non-English languages or code.
Why is my token count higher than I expected? System prompts count toward your input tokens and are often larger than people realize. A detailed system prompt with role instructions, formatting rules, and few-shot examples can easily be 500-2,000 tokens before you add any user message. If your costs seem higher than expected, check whether you’re accounting for your full system prompt in your estimates.
Does whitespace count as tokens? Yes. Whitespace characters — spaces, tabs, newlines — are part of the token stream. Leading spaces are often merged with the following word into a single token. Unnecessary double-spacing, trailing spaces, and extra blank lines all add to your token count, though the overhead is usually minor.
What’s the most token-efficient format for passing structured data?
Markdown tables are generally more token-efficient than JSON for structured data that a language model will read. A table row like | Product A | $12.99 | In stock | is more compact than the equivalent JSON object. CSV is even more compact for large datasets. Use JSON only when structure-preservation in the output matters.
How do I estimate token costs for a new AI project before I build it? Start with a representative sample of real inputs — 10-20 examples that reflect your actual data range. Tokenize them, calculate average tokens per request, multiply by your expected daily call volume, and project to monthly and annual costs. Account separately for input tokens (your prompt plus context) and output tokens (the model’s response), as they’re billed at different rates.
Related reading
- AI Token Counter — exact token counts and cost estimates for any model
- AI Batch API Discount Guide — cut your per-token cost by 50%
- AI Cost Projection and Budgeting Framework
