How to Count Tokens for LLM Prompts: A Practical Token Counter Guide

Every time you send text to a language model, the first thing that happens before any reasoning is tokenization. The model does not read characters or words the way you do. It chops your text into tokens, and then everything downstream — the price you pay, whether your prompt fits in the context window, how long the reply takes — is measured in those tokens, not in characters or sentences. If you build anything on top of an LLM, getting comfortable with token counting is the single cheapest way to stop being surprised by your bill.

This guide explains what a token actually is, why the count matters so much, how it shifts across models, and how to trim a prompt down without losing meaning. You can follow along and check any text against a live estimate using the AI Token Counter.

What a Token Actually Is

A token is a sub-word chunk. The tokenizer learns a vocabulary of frequent character sequences and then encodes your text as a sequence of those chunks. Common short words like "the" or "and" usually become a single token. Longer or rarer words get split: "tokenization" might land as "token" + "ization", and an unusual proper noun can fragment into three or four pieces.

The rule of thumb worth memorizing is this: in English, one token averages roughly 4 characters, or about 0.75 words. That means 1000 tokens is approximately 750 words, and a 750-word article is around 1000 tokens. Whitespace, punctuation, and a leading space all get folded into tokens too, which is why a space before a word often rides along inside the same token rather than counting separately.

A few things skew the average. Code, JSON, and markup tokenize more densely because of all the brackets, indentation, and symbols. Non-Latin scripts are far heavier: a single Chinese character frequently maps to one token on its own, and sometimes splits into two or three sub-word pieces, so a 1000-character Chinese prompt can bill closer to 1500 to 2000 tokens rather than the 250 you might naively guess from the 4-characters-per-token rule.

Why Token Count Drives Cost and Context

Two hard limits in every LLM workflow are denominated in tokens.

The first is cost. Vendors price by the million tokens, and they bill input and output separately — usually with output costing more. As of 2026, public list rates put GPT-4 Turbo at $10 per million input tokens and $30 per million output, Claude Sonnet 4 at $3 / $15, Claude Haiku 4.5 at $1 / $5, and Gemini 2.0 at $0.5 / $1.5. A workflow that runs the same 2000-token prompt ten thousand times a day is moving 20 million input tokens daily before a single word of output is generated. At GPT-4 Turbo rates that is $200 a day just on input.

The second is the context window. Every model has a maximum number of tokens it can hold at once — system prompt, conversation history, retrieved documents, your new message, and the reply all share that budget. When you stuff a long document into a retrieval-augmented prompt, you are spending context tokens. Run over the limit and the request is rejected or the oldest history gets silently truncated. Counting tokens up front tells you whether your prompt fits before you find out the hard way.

The output side deserves a specific warning, because it causes most billing surprises. People size the prompt carefully and forget the reply is billed too, often at a higher per-token rate. A 500-token question that triggers a 3000-token code-generation answer costs far more on the output line than the input line.

How Counts Differ Across Models

Each vendor trains its own tokenizer, so the same text produces different counts.

GPT-4's cl100k_base encoding averages about 4 characters per token on English. Anthropic's tokenizer runs leaner, around 3.5 characters per token, which means Claude often reports more tokens for the same English text. That sounds like bad news until you check the price: Claude Sonnet 4's lower per-token input rate ($3/M versus GPT-4 Turbo's $10/M) usually makes the total cheaper even with the higher token count. Gemini uses a documented per-character rate that lands in a similar ballpark.

The practical takeaway: never assume a token count from one model transfers to another, and never reason about cost from token count alone. Compare total dollars for the same input across model families. A long-context task that looks expensive on one vendor can be markedly cheaper on another once both token count and per-token price are accounted for. A side-by-side estimate, like the one in the LLM pricing calculator, makes that comparison concrete instead of guesswork.

A Worked Example

Take this sample paragraph:

"Our support bot reads the customer's last three messages, looks up their order status, and drafts a polite reply suggesting the next step."

Count it: that is 24 words and 138 characters including spaces. Apply the word-based estimate (words ÷ 0.75): 24 ÷ 0.75 ≈ 32 tokens. Apply the character-based estimate (characters ÷ 4): 138 ÷ 4 ≈ 35 tokens. The two heuristics bracket the real figure, which for this prose lands around 30 tokens on GPT-4 — the methods agree within a token or two, which is exactly the precision you want for budget planning.

Now scale it. If that paragraph is the template and you fire it 50,000 times a day with a similar-length reply, you are looking at roughly 30 input + 30 output = 60 tokens per call, or 3 million tokens a day. At Claude Haiku 4.5 rates ($1 input / $5 output per million), that is about $0.09 input plus $0.45 output, so well under a dollar a day. The same volume on GPT-4 Turbo would be $0.30 + $0.90, roughly $1.20 a day. The arithmetic is trivial once you have the token count; the whole job is getting that count right.

Trimming Prompts Without Losing Meaning

When a prompt is too expensive or too close to the context limit, here is where I look first. In my own work the biggest single saving has consistently come from system prompts that grew over months — I once cut a 1400-token instruction block to 600 tokens just by deleting three paragraphs of redundant examples the model never needed, and quality did not move.

Concrete tactics that pay off:

Cut redundant instructions. Models rarely need the same rule stated three ways. Say it once, clearly.
Drop few-shot examples that no longer earn their keep. Each example can be hundreds of tokens. Test whether two examples work as well as five.
Summarize history instead of replaying it. In a long chat, a 100-token running summary often beats 3000 tokens of verbatim transcript.
Compress retrieved documents. Pull the relevant passages, not whole files.
Watch your output cap. Setting a sane max-output limit prevents a runaway reply from ballooning the bill.

Before and after each edit, count the tokens so you know the saving is real. If your text is plain prose, a fast word counter gives you a quick word total that you can convert to a token ballpark with the 0.75 multiplier, then confirm against a proper estimate.

Putting It Together

Token counting is not an advanced topic — it is the unit of account for the entire LLM economy. A token is a sub-word chunk; English runs about 4 characters or 0.75 words per token, so 1000 tokens is roughly 750 words; cost and context windows are both measured in tokens; and counts plus prices both shift across GPT, Claude, and Gemini. Once you can estimate a count in your head and verify it in a second, sizing a prompt, comparing vendors, and explaining a cost spike all stop being mysteries. Paste your real prompts into the AI Token Counter, watch the numbers, and trim from there.

Made by Toolora · Updated 2026-06-13