Snoika
Glossary·How AI search engines work

What are LLM tokens

Definition

LLM tokens are the small pieces of text that a large language model (LLM) reads and writes.

A token can be a whole word or part of a word. Spaces and punctuation marks can also be tokens. The model never sees words the way you do. If you're asking what are LLM tokens, the practical answer is that they're the unit AI tools use to measure text for pricing and limits on how much they handle.

TermLLM tokens3 min read · reviewed by the Snoika team

LLM tokens defined for marketers

For a marketer, a token is the unit an AI assistant uses to measure your content. Word counts don't apply here. OpenAI's own rule of thumb puts one token at about 4 characters, so 100 tokens equal roughly 75 words of English. The ratio changes with the language and the model. Text in German or Japanese takes more tokens for the same meaning, as does text full of code. Before you budget a prompt, paste the text into the provider's tokenizer and check the real count.

Why tokens shape AI visibility

Tokens decide how much of your page an AI assistant can read and how much it can quote. Assistants that search the web split pages into passages measured in tokens. Then they pick the passages that answer the question best. A fact stated plainly in two sentences fits inside one passage and is easy to cite. The same fact spread across a long, padded section gets split apart or cut off. Brand names matter too. A rare or invented name breaks into several odd fragments, and the model has less experience with those fragments as a sequence. Check that your key claims sit close to the headings they answer.

How tokenization works

Tokenization turns raw text into numbers the model can process. Most current models use a method called byte pair encoding (BPE), which learns common character sequences from training data.

  1. 011The tokenizer splits the text into known pieces from a fixed vocabulary.
  2. 022It maps each piece to an ID number.
  3. 033The model reads the IDs and predicts the next token, one at a time.
  4. 044The tokenizer turns the predicted IDs back into readable text.

What one token looks like

A common short word such as "the" is one token, with its leading space included. Longer or rarer words split up. A tokenizer breaks "unbelievable" after "un" and "believ". The final piece is "able". Capital letters and spaces change the split as well, so "Apple" and " apple" can get different IDs. Numbers break into chunks of 1 to 3 digits.

Token limits explained

Every model caps how many tokens it can handle at once. Marketers need to know 2 limits.

Context window

The context window is the total number of tokens the model can hold in one exchange. That total covers your prompt and any pasted documents. The conversation history and the reply count toward it too. OpenAI's GPT-4o has a 128,000-token context window, which is roughly 300 pages of text. When a chat runs past the window, the oldest material drops out or gets compressed, and the model stops "remembering" your early instructions.

Output limit

The output limit is the most the model writes in a single reply, and it's much smaller than the context window. GPT-4o stops at 16,384 output tokens. Ask for a 15,000-word report and the reply will end early, mid-sentence. Split long jobs into sections, and check each provider's model page, because these limits change with new releases.

How tokens affect AI output

Token counts drive the cost and speed of every AI response. They also shape its quality. API providers charge per token, and output tokens cost more than input tokens. Long prompts slow responses down. Bloated prompts also bury your instructions among filler the model has to weigh. Short, specific prompts give sharper answers for less money. Watch the usage figures in your tool's dashboard to see where the tokens go.

Related AI glossary terms

Tokenizationthe process that splits text into tokens. The token is the result.
Context windowthe token budget for one exchange. Tokens are the units counted inside it.
Embeddingsnumber lists that capture a token's meaning, which differs from the token's ID.
Large language model (LLM)the system that reads and predicts tokens.

The next step is to run one of your own pages through a free tokenizer and compare the token count with the word count. That one test shows what are LLM tokens in practice and how much of your content an assistant takes in.

Look up more AI search and SEO terms in our glossary.

Related terms

All terms
AI OverviewsAn AI Overview is a short answer, written by AI, that Google shows above its regular search results. It's a paragraph or a short list built from several web pages, with source cards that link back to them. Google writes each summary at the moment of the search and shows one only when it judges a sumMCPMCP, short for Model Context Protocol, is an open standard that lets an AI application connect to outside tools and data through one shared interface. Anthropic released it in November 2024. So when people ask what is MCP, the plain answer is a common plug. It lets an assistant such as Claude read a company's files or call its services without custom code for each connection.AI assistantAn AI assistant is software that answers requests written in everyday language and completes tasks for you. It runs on a large language model (LLM), which reads your prompt and writes the reply. If you're asking what is an AI assistant, you've probably used one already. ChatGPT is one, and so is GooSearchGPTSearchGPT is OpenAI's AI search tool. It answers questions in plain language and cites the web pages it used. OpenAI announced the prototype on July 25, 2024 and later built it into ChatGPT. ChatGPT search is SearchGPT under its current name.AI ModeGoogle AI Mode is a conversational search mode inside Google Search. It answers a question with one written, cited response in place of a page of links. It runs on a custom Gemini model and handles long questions with several parts, plus folSGEGoogle SGE (Search Generative Experience) was Google's test of AI-written answers at the top of its search results. It opened as an opt-in experiment in Search Labs in May 2023. A year later, Google renamed it AI Overviews and switched it on for every US searcher. The name is gone, but the feature i

See how AI talks about your brand

Get free audit