o200k_base vs cl100k_base explained

If you have counted tokens programmatically you have met the names: cl100k_base, o200k_base, and before them p50k_base and r50k_base. They are encodings — tiktoken’s word for a complete tokenizer specification, consisting of three things:

1. a vocabulary of byte sequences,
2. the merge ranks that say which pairs to combine and in what order,
3. a regex split pattern applied before any merging, which decides where a chunk can begin.

Two models with the same encoding produce identical token ids for identical text. Two models with different encodings can differ by 20% or more on the same input. So the first question in any token-counting exercise is which encoding you are actually targeting.

The lineage

Encoding	Vocabulary	Era
r50k_base	~50,000	GPT-3 (davinci and friends)
p50k_base	~50,000	Codex, text-davinci-002/003
cl100k_base	~100,000	GPT-3.5-turbo, GPT-4, text-embedding-3-*
o200k_base	~200,000	GPT-4o onward — and everything since

The jump from p50k_base to cl100k_base added tokens for runs of whitespace, which is why counting indented code with a GPT-3-era tokenizer over-counts badly. The jump to o200k_base doubled the vocabulary again.

o200k_base is now the answer for essentially the whole current OpenAI line-up. tiktoken’s model registry maps the entire gpt-5*, gpt-4.1* and o-series prefix space to it, so you do not need a per-model lookup — a prefix match is enough. There is also a newer registry entry, o200k_harmony, which reuses o200k_base’s ranks and split pattern and only adds control tokens. For counting ordinary text the two are interchangeable.

What doubling the vocabulary buys

A larger vocabulary means longer common substrings survive as single tokens, so the same text encodes to fewer of them. The effect is small for plain English — that was already well covered at 100,000 entries — and substantial everywhere else:

• Non-English text gains the most. The extra 100,000 slots went disproportionately to sequences in scripts that cl100k_base had to spell out in fragments. Chinese, Japanese, Korean, Hindi, Arabic and Cyrillic all encode meaningfully shorter.
• Code gains from longer identifier and keyword fragments, and from more whitespace runs.
• Structured data gains from common key names and punctuation clusters becoming single tokens.

Fewer tokens for the same text is a direct saving in two currencies at once: money, because you are billed per token, and context, because your window is measured in them. This is why counting a Japanese prompt with cl100k_base when the model actually uses o200k_base overstates the cost — sometimes by a wide margin — while the same mistake on an English paragraph is barely visible.

What it costs

Rank data is not small. o200k_base is roughly 2.3 MB; cl100k_base roughly 1.1 MB. Doubling the vocabulary roughly doubled the file.

For a server that is a non-issue. For a browser-based counter it is the whole engineering problem, and it is why the token counter on this site never loads either file up front: the ranks are behind a dynamic import, split into their own chunk, and nothing is fetched until you actually tokenize something. The page itself ships a few kilobytes of JavaScript. Once loaded, the counting happens locally — no request carries your text anywhere.

Which one do you need?

Provider	Models	Encoding	Count is
OpenAI	GPT-5.6, GPT-5.5, GPT-5.4, GPT-5.1, GPT-4.1, GPT-4o	o200k_base	exact
OpenAI (legacy)	GPT-4, GPT-3.5-turbo, text-embedding-3-*	cl100k_base	exact
Anthropic	Claude Fable / Opus / Sonnet / Haiku	not published	estimated
Google	Gemini 3.x, Gemini 2.5	not published	estimated
xAI	Grok 4.x	not published	estimated
DeepSeek	V4 Pro, V4 Flash	not published	estimated
Meta (hosted)	LLaMA 3.3 70B	not published locally	estimated

If you are calling anything current from OpenAI, you want o200k_base. You want cl100k_base only for the older chat models and the text-embedding-3 family — which is a real case, since embedding pipelines tend to outlive the chat models they were built alongside.

Why the rest of the table says “estimated”

Anthropic, Google, xAI and DeepSeek do not publish a vocabulary file you can run locally. xAI exposes a /tokenize-text endpoint, which is useful but is a network call against your live text rather than a local tokenizer. Meta ships weights, but the hosted endpoints most people use do not expose a counting API.

So for those models, any local counter is applying a heuristic. This one counts roughly four characters per token for Latin script and roughly 1.5 for CJK and Hangul, then labels the result as an estimate rather than presenting it as a bill. It is close enough for capacity planning and should not be used to reconcile an invoice.

One case deserves a specific warning: Anthropic changed tokenizers mid-family. Claude Opus 4.7 and later produce roughly 30% more tokens for the same text than 4.6 and earlier. No character-count heuristic can detect that, so a single estimate cannot be right for both halves of the family. If you are budgeting Claude spend at volume, build in margin, or measure against the provider’s own count-tokens API.

Practical rules

• Match the encoding to the model you will actually call. A cl100k_base count against an o200k_base model over-counts, worst on non-English text and code.
• Do not mix encodings inside one pipeline’s accounting. If a retrieval step embeds with cl100k_base and a generation step runs on o200k_base, those two token counts are not the same unit and should not be summed.
• Treat estimated counts as a range. For Anthropic, Google, xAI and DeepSeek, plan with a margin rather than a point figure.
• Look at the split, not just the total. The visualizer on the token counter renders the real BPE boundaries, which is the quickest way to see that your identifiers or your JSON keys are fragmenting.
• Then price it. The model pricing comparison has input, cached input and output rates side by side, so you can see what a 20% counting difference is actually worth on your volume.

Encodings are one of the few parts of this stack that are completely deterministic and fully documented. Getting them right costs one lookup and removes a whole class of budgeting error.