How It Works

This tool tokenizes your text using the actual tokenizers for each model, entirely inside your browser. Your text is never sent to a server.

• GPT-4o: tokenized synchronously using gpt-tokenizer's pure-JS o200k_base implementation.
• Claude 3.5 Sonnet: tokenized using Anthropic's published vocabulary (claude.json) with tiktoken's WASM engine — loaded asynchronously in the background.
• Gemini 2.0 Flash: Google has not released their tokenizer as a standalone library; token count is estimated at ×1.15 from GPT-4o based on benchmarks for code and JSON payloads.

What is a token?

Large language models don't process raw characters; they work on tokens — variable-length chunks that the model treats as single units. For English prose, one token is roughly 4 characters or ¾ of a word. Spaces, punctuation, numbers, and code symbols each have their own rules.

The coloured spans in the visualizer above mark individual tokens. Each colour change is a token boundary. You'll see that short common words are usually one token, while longer or less common words are split into two or more.

Why token counts differ across models

Every provider ships its own tokenizer with a different vocabulary size and merge rules. The same 1,000-word document can cost 20–30% more on one API than another, purely because of how each tokenizer encodes it — especially for code, JSON, and non-English text.

Claude's actual token counts are computed in-browser using Anthropic's published @anthropic-ai/tokenizer vocabulary. Gemini's tokenizer is not publicly available — its count remains an estimate. Differences are smallest for plain English prose and largest for heavily indented code or JSON.

Why token counts matter

• Context windows: Every model has a maximum token limit. Sending more tokens than the limit truncates your input or raises an API error.
• API costs: Cloud AI APIs charge per token, so a 10,000-token prompt costs 10× more than a 1,000-token one.
• Latency: Inference time scales roughly with token count — fewer tokens means faster, cheaper responses.

Model pricing (Q1 2025)

Implementation

GPT-4o: tokenized synchronously using gpt-tokenizer, a pure JavaScript BPE implementation bundling the full o200k_base vocabulary — no WASM, no server round-trip.

Claude 3.5 Sonnet: tokenized using Anthropic's published @anthropic-ai/tokenizer vocabulary (claude.json, loaded from jsDelivr) with tiktoken's Rust-compiled WASM engine. Both resources are verified with SRI integrity hashes and loaded asynchronously in the background so they don't delay the initial page render.

Privacy

All processing runs locally. Sensitive text — API keys, personal data, proprietary code — never leaves your device.