Every time you pay an AI API bill, you’re paying for tokens — but most developers and practitioners can’t precisely define what a token is. That gap creates real problems: overestimated context capacity, budget surprises, and prompts that hit length limits without warning.
What a Token Actually Is
A token is the smallest unit of text that a language model processes. It is not a character, not a word, and not a syllable — though tokens often look like pieces of words.
When text enters a language model, a component called a tokenizer converts it into a sequence of integers. Each integer corresponds to a token — a text fragment that exists in the model’s vocabulary. The model never sees your raw text directly; it sees a list of numbers that map back to text fragments.
For typical English prose, one token corresponds to approximately 0.75 words, or about 4 characters. That rule of thumb is useful for rough estimates, but the actual tokenization varies considerably depending on what you’re writing. A simple sentence like “The meeting starts at 9 AM” might tokenize as: ["The", " meeting", " starts", " at", " 9", " AM"] — 6 tokens for 7 words, close to the 0.75 ratio. But token counts for code, non-English text, and special characters diverge significantly.
The vocabulary size matters too. OpenAI’s tiktoken (used for GPT-4o and GPT-5) uses a vocabulary of approximately 100,000 tokens. Anthropic’s tokenizer (used for Claude) is similar in size. These large vocabularies allow common English words to map to single tokens, while rare words or foreign text get split into multiple sub-word tokens.
How Different Content Types Tokenize
Understanding tokenization differences across content types explains a lot of counterintuitive API cost behavior.
Standard English prose: High token efficiency. Common words like “the”, “is”, “at”, “of” each map to a single token. A 500-word paragraph of business writing typically uses around 650–700 tokens.
Technical and uncommon vocabulary: Lower efficiency. Words like “cryptocurrency”, “immunotherapy”, or “photolithography” often split into multiple tokens. “cryptocurrency” might tokenize as ["crypto", "currency"] — two tokens for one word. Jargon-heavy content routinely runs 20–30% higher than standard English token ratios.
Code: Significantly more tokens per character than prose. Code uses indentation, special characters, variable names, and syntax that tokenizers don’t handle as efficiently as natural language. A 100-line Python function might tokenize to 400–700 tokens depending on complexity, comment density, and variable naming style. Long, descriptive variable names are more expensive than short ones — not an argument for cryptic naming, but worth knowing.
JSON: Usually inefficient. JSON structure characters ({, }, [, ], :, ") each consume tokens. Pretty-printed JSON with indentation costs 20–30% more than compact JSON. A well-designed API that receives large JSON payloads should strip formatting before sending to an LLM.
Non-Latin scripts and multilingual text: Often the most token-expensive per visible character. Chinese, Japanese, Korean, Arabic, and other non-Latin scripts frequently tokenize at 2–4 tokens per character rather than 4 characters per token. This means a 100-word Chinese text may cost 3–5× as many tokens as 100 words of English. This has real cost implications for applications serving non-English users.
Numbers and dates: Variable. Short numbers tokenize efficiently; long numeric strings may split unexpectedly. “2026” is typically one token. A long phone number or product ID might tokenize character by character.
Tokens vs Context Windows
The context window is the maximum number of tokens a model can process in a single request — input plus output combined. Understanding this limit is essential for designing prompts and workflows.
GPT-4o has a 128,000-token context window. Claude’s Sonnet and Opus 4 models support 200,000 tokens. Gemini 2.0 Flash and Pro support up to 1 million tokens (with 2M available in some configurations). These are large numbers, but they fill up faster than most people expect.
Consider a RAG-based document assistant: system prompt (500 tokens) + retrieved document chunks at 10 chunks × 1,500 tokens each (15,000 tokens) + conversation history at 10 turns × 600 tokens (6,000 tokens) + user question (100 tokens) = 21,600 tokens before the model generates a single word. At $5 per million GPT-4o input tokens, each query costs about $0.108 — in the context of thousands of daily users, that adds up quickly.
The context window also affects what happens when you exceed it: the API returns an error or (in some implementations) silently truncates the oldest content. Knowing your average prompt size as a token count — not a word count — lets you plan for this before it causes problems in production.
How Tokens Connect to API Costs
Every major AI API charges by the token. Understanding the billing mechanics prevents surprises.
Most providers bill input and output tokens separately, with output tokens costing 2–5× more than input tokens. The rationale: generating tokens is computationally more expensive than reading them. This asymmetry means the cost structure rewards concise output — a model configured to write 1,000-word essays by default costs significantly more than one configured to write 200-word summaries with equivalent information density.
Billing is also per token, not per word or character. If your prompt contains 743 tokens, you’re billed for exactly 743 tokens — partial token billing doesn’t apply. The granularity matters at scale: a 50-token system prompt reduction across 100,000 daily calls saves 5 million input tokens per day.
To see exactly how many tokens your prompts consume before you run them, paste your text into the free AI Token Counter — it returns the exact token count for GPT-4o, GPT-3.5, and Claude tokenization schemes, plus a cost estimate at your specified call volume.
Why Tokenization Differs Between Models
Not all AI models use the same tokenizer, and the differences matter when you’re switching between providers.
OpenAI’s tiktoken (used for GPT-4o, GPT-5, and GPT-3.5) and Anthropic’s tokenizer for Claude produce slightly different token counts for the same text — typically within 5–15% of each other for English prose, but diverging more for code and non-Latin languages.
If you’re running the same workflow on both OpenAI and Anthropic models and comparing costs, use the actual tokenizer for each. Counting OpenAI tokens and applying them to Claude’s pricing (or vice versa) introduces systematic error in your cost projections.
This also matters for context window calculations. If you’re managing conversation history to stay under a context limit, a token count from one tokenizer may undercount the other model’s actual usage. The safest approach is to use each model’s official tokenizer library — tiktoken for OpenAI, Anthropic’s token counter for Claude.
Count Your Tokens Before Running Prompts
The gap between estimated token count and actual token count is where API budget surprises happen. Before you build a workflow at scale, measure the token footprint of your actual prompts. Our free AI Token Counter shows the token count for any text you paste in — English, code, JSON, or multilingual — along with a side-by-side cost estimate for GPT-4o, GPT-4o mini, and Claude Sonnet at your expected call volume. It’s the fastest way to turn a gut estimate into a real number.
Frequently Asked Questions
Is 1 token always equal to 4 characters? That’s a rough average for standard English text, not a precise rule. Common English words tend to tokenize efficiently, often mapping to a single token per word. But code, numbers, non-Latin scripts, and uncommon vocabulary can tokenize at 1–2 tokens per character. For accurate counts, use the actual tokenizer rather than the character approximation.
Does the model charge tokens for whitespace and punctuation? Yes. All characters in your prompt, including spaces, newlines, and punctuation, are tokenized and billed. Extra whitespace and unnecessary formatting characters add to your token count without adding semantic value.
How many tokens can I fit in GPT-4o’s context window? GPT-4o supports a 128,000-token context window for the combined input and output. A 128,000-token context window holds approximately 96,000 English words — equivalent to a short novel. In practice, very long contexts also increase latency and, at the extreme end of the window, can affect the model’s ability to retrieve information from early in the context.
Do images count as tokens? Yes, for vision-capable models. Images are converted to tokens at a rate that depends on image size and detail level. A 512×512 image typically costs 170–340 tokens. High-resolution or detailed images can cost 1,000+ tokens. This is why image-heavy applications need to account for visual token consumption, not just text.
Why do I sometimes get different token counts from different tools? Different tools may use different tokenizer versions. OpenAI’s tiktoken library is the authoritative source for GPT models. Anthropic’s official count tool is authoritative for Claude. Third-party token counters may approximate or lag behind tokenizer updates. For production cost estimates, use the official library or a tool that wraps it directly.