What Is a Token in AI? Essential Guide Behind Every Costly Prompt

Your AI response cuts off mid-sentence. Your monthly API bill comes in higher than expected, even though you didn’t send more messages. A long conversation suddenly seems to “forget” something you said earlier. All three of these have the same root cause, and understanding it will make every AI tool you use make a lot more sense.

That root cause is tokens. This guide explains what a token in AI actually is, how tokenisation works under the hood, why the same sentence can cost different amounts depending on which model you’re using, and how tokens connect to the training process that builds these models in the first place.

What Is a Token in AI?

A token is the smallest unit of text an AI language model actually reads, processes, and generates. It’s not a word, and it’s not a character; it’s something in between, called a subword unit, determined by an algorithm the model’s creators built specifically for that model.

Before a model can do anything with your prompt, it has to convert your text into these units, then into numbers, since a neural network can only do math, not read English directly. That conversion process is called tokenisation, and it happens before every single request, whether you’re chatting with ChatGPT, asking Claude a question, or building an app on top of an API.

A rough rule of thumb for English text: one token equals about four characters, or roughly three-quarters of a word. That means 100 tokens get you approximately 75 words, and a typical page of English prose runs somewhere around 500 to 600 tokens.

Tokens vs. Words vs. Characters

This distinction trips up more people than any other part of understanding tokens, so it’s worth seeing in practice.

Take the sentence “The quick brown fox jumps.” Most tokenisers split that into something close to six tokens: “The”, ” quick”, ” brown”, ” fox”, ” jumps”, “.” Common English words usually map to a single token each.

Now take an uncommon word, like “tokenisation” itself. A tokeniser without that exact word in its vocabulary will break it into smaller pieces, something like “token” and “ization,” two tokens instead of one. Made-up words, typos, and rare proper nouns get split even further, sometimes into individual syllables or characters.

This has a real, measurable consequence for anyone using AI in a language other than English. Because most major tokenisers were built primarily around English-language training data, the same sentence in Portuguese or Indonesian can require 1.5 to 1.6 times as many tokens to represent, which means non-English users often pay more and hit context limits faster for equivalent content, a genuine equity and cost issue that most beginner explanations skip entirely.

Code, JSON, and mathematical notation are similarly token-heavy, since punctuation-dense text with lots of unique symbols doesn’t compress as efficiently as plain prose does. A 200-line Python function can easily consume 800 to 1,200 tokens, more than the word count alone would suggest.

How Tokenisation Actually Works

Most modern AI models, including GPT, Claude, and Llama, use a technique called Byte Pair Encoding, or BPE, to build their tokeniser’s vocabulary. It’s worth understanding the mechanics, because it explains some genuinely strange AI behaviour you may have encountered.

BPE starts by treating text as individual characters. It then scans a massive training corpus and finds the most frequently occurring pair of adjacent characters or symbols, merges that pair into a single new token, and repeats the process thousands of times until it reaches a target vocabulary size, typically somewhere between 50,000 and 200,000 tokens depending on the model.

The result is a vocabulary where common patterns, whole words, common prefixes and suffixes, and frequent punctuation combinations each get their own single token, while rare or unusual text gets broken into smaller fragments on the fly.

A Strange Side Effect: Glitch Tokens

Because BPE builds its vocabulary purely from frequency in a training corpus, separate from how well a model actually learns to use each token during training, it can accidentally create what researchers call “glitch tokens.” A well-known example: a Reddit username appeared often enough in a tokeniser’s training corpus to earn its own dedicated token, but rarely enough in the model’s actual training conversations that the model never properly learned what that token meant. Asking early versions of ChatGPT to simply repeat that specific string back produced completely unrelated, nonsensical output instead.

Researchers have continued documenting this phenomenon into 2026, finding that a small but meaningful percentage of tokens across major model vocabularies remain effectively undertrained in this way. It’s a good reminder that tokenisation isn’t a purely mechanical, side-effect-free process; it’s a real design choice with real downstream quirks.

Why Different AI Models Tokenise Text Differently

This is a genuinely common point of confusion: the same sentence does not cost the same number of tokens across different AI providers, because each one uses its own tokeniser, trained on its own data with its own target vocabulary size.

OpenAI’s models use a tokeniser called tiktoken, with the specific vocabulary varying by model generation: cl100k_base for GPT-4-era models, o200k_base for newer GPT-4o and GPT-5 family models. Anthropic’s Claude models use a separate, proprietary tokeniser, and Anthropic doesn’t publish a standalone local tokeniser the way OpenAI does, so counting Claude tokens accurately requires using Anthropic’s own token-counting endpoint rather than approximating with a different provider’s tool. Meta’s Llama models and Google’s Gemini models each use their own SentencePiece-based or tiktoken-style vocabularies as well.

The practical takeaway: never assume a token count from one model’s tokeniser applies to another. If you’re building anything that estimates cost or manages context length across multiple providers, you need to count tokens using each provider’s specific tokeniser, not a universal estimate.

Tokens and Context Windows

A model’s context window is the maximum number of tokens it can hold in memory at once, counting your prompt, any earlier messages in the conversation, and the response it’s generating, all combined.

This is why very long conversations sometimes seem to lose track of something you said early on. Once a conversation’s total token count approaches the model’s context limit, earlier tokens may get truncated or summarised to make room, depending on how the specific application manages that overflow.

It’s also why an AI response occasionally cuts off mid-sentence: if you’ve set a maximum output token limit, or if the combined conversation is close to the model’s ceiling, the model simply runs out of room to finish its thought.

How Token Pricing Actually Works

If you’ve used an AI tool through an API rather than a consumer chat app, you’ve likely seen pricing listed as a rate “per million tokens,” split separately into input and output pricing.

As of mid-2026, pricing across major providers varies widely by capability tier. Budget-oriented models like Gemini Flash-Lite or GPT-5 nano can run as low as roughly $0.05 to $0.15 per million input tokens, while frontier reasoning models can run $5 to $20 or more per million input tokens, with output tokens typically priced two to six times higher than input tokens across virtually every provider.

That output-token premium matters more than people realise. A request with a short prompt but a long, detailed response can cost significantly more than a longer prompt that generates a brief answer, since you’re paying more per token for what the model generates than for what you send it.

A practical example: a simple customer support query, roughly 1,000 input tokens and 500 output tokens, might cost well under a cent on a budget-tier model, but the same request pattern run through a premium frontier model, or run millions of times a month at scale, is exactly where token costs become a serious line item on an engineering budget rather than a rounding error.

What Is AI Training and How Do Tokens Fit In?

Tokens aren’t just relevant to using a finished AI model; they’re the fundamental unit the model is built from in the first place.

AI training, specifically for large language models, works by feeding a model enormous amounts of text, broken down into tokens using the same tokenisation process described above, and having the model repeatedly predict the next token in a sequence. It sees part of a sentence, guesses what token comes next, checks that guess against the real answer, and adjusts its internal parameters slightly based on how wrong it was. Repeated across trillions of tokens, and often across many passes over that training data, this simple prediction task is what teaches a model grammar, facts, reasoning patterns, and coding ability.

Modern frontier models are typically trained on corpora measured in the trillions of tokens, drawn from sources like books, websites, code repositories, and licensed datasets. The size of that training corpus, combined with the number of parameters in the model and the amount of computing power used, are the three factors researchers generally point to as driving how capable a resulting model turns out to be.

This is also why the tokeniser itself is chosen before training even begins, and essentially locked in afterwards. Changing a model’s tokeniser after training would mean every pattern it learned about which tokens tend to follow which others no longer lines up correctly, so tokeniser design is one of the first, most consequential decisions made when building a new model.

Tips to Manage Token Usage

Write plain, direct prompts. Prose is the most token-efficient input format. Unnecessary preamble, repeated instructions, and overly formatted requests all add tokens without adding value.

Watch output-heavy tasks specifically. Since output tokens cost more than input tokens across nearly every provider, tasks that generate long responses, like full document drafts, cost more per request than you might estimate from prompt length alone.

Use a smaller model for simple tasks. Routing straightforward requests, classification, formatting, short lookups, to a budget-tier model and reserving frontier models for genuinely complex reasoning is one of the most effective ways to cut costs at scale.

Count tokens with the right tool. Use each provider’s own tokeniser or counting endpoint rather than assuming a universal word-to-token ratio, especially if you’re comparing costs across different AI providers.

Common Mistakes People Make About Tokens

Assuming 1 token always equals 1 word. It’s closer to three-quarters of a word for English on average, but that ratio shifts significantly for other languages, code, and uncommon vocabulary.

Comparing token counts across different models directly. The same text tokenises differently depending on which provider’s tokeniser processed it. A “shorter” prompt on one model’s counter isn’t necessarily cheaper on a different provider.

Forgetting that conversation history counts too. In a multi-turn conversation, every previous message typically gets resent as part of the context on each new request, which means token usage, and cost, grows with conversation length, not just with your latest message.

Ignoring output token pricing. Many people budget based on input costs alone and get surprised by bills from tasks that generate long, detailed responses.

Frequently Asked Questions

What is a token in AI, in simple terms?

A token is a small chunk of text, roughly three-quarters of a word on average for English, that an AI model reads and generates one piece at a time, rather than processing whole sentences or documents directly.

What are AI tokens used for besides chatting?

Tokens determine a model’s context window limit, how conversation history is tracked, and how API usage is priced and billed, making them the basic unit of both a model’s memory and its cost.

Are tokens the same across ChatGPT, Claude, and Gemini?

No. Each provider uses its own tokeniser with its own vocabulary, so the same sentence can produce a different token count on different platforms. Always use a provider’s own tokeniser or counting tool for accurate estimates.

What is AI training, and how many tokens does it involve?

AI training is the process of teaching a model to predict the next token in a sequence, repeated across massive amounts of text. Modern frontier models are typically trained on corpora measured in the trillions of tokens.

Why do I pay more for AI output than input?

Generating new text requires more computation per token than reading existing text does, which is why nearly every major AI provider prices output tokens at two to six times the rate of input tokens.

Do longer words always use more tokens?

Not necessarily. Common long words often map to a single token if they appear frequently in the tokeniser’s training data, while short but unusual words or made-up terms can get split into multiple tokens.

For a deeper technical look at the algorithm underlying most modern tokenisers, Wikipedia’s entry on byte pair encoding covers its origins as a 1994 data compression technique, long before it became the foundation of how today’s AI models read text.

Conclusion

Tokens are the quiet unit underneath almost everything about how modern AI works: what it costs, how much it can remember in one conversation, and even the strange behaviour it occasionally shows on rare or unusual text. Understanding tokens won’t make you a machine learning engineer, but it will make AI pricing, context limits, and the occasional weird glitch make a lot more sense the next time you run into one.

If you take one thing from this guide, let it be this: tokens aren’t words, they aren’t universal across providers, and they’re worth paying attention to any time cost, memory, or unexpected AI behaviour is part of the conversation.