What is type-token ratio in text analysis
Type-token ratio (TTR) measures how varied a text's vocabulary is.
You divide the number of distinct words (types) by the total number of words (tokens). A score near 1 means almost every word is new. A score near 0 means the same words keep coming back. Linguists use the type-token ratio as the simplest measure of lexical diversity.
Type-token ratio definition
A type-token ratio is the share of a text's words that are unique. A token is any single word as it occurs. A type is a distinct word form, however often it's repeated. In the sentence "the dog saw the cat", there are 5 tokens but only 4 types, because "the" appears twice. Before you trust a score, check how the tool defines a word. Some tools treat "Run" and "run" as one type and others don't. Lemmatized counts also merge "run" and "ran", which lowers the score.
How to calculate it
TTR is the number of types divided by the number of tokens. If you want a percentage, multiply by 100. Any TTR calculator does the same arithmetic, so the result depends on how the text is cleaned first.
The type-token ratio formula step by step
- 011Lowercase the text and strip out punctuation.
- 022Split it into words and count them. That count is your tokens.
- 033Count each distinct word once. That count is your types.
- 044Divide types by tokens.
A worked example
Take "The dog chased the cat and the cat ran up the tree." It has 12 tokens. "The" appears 4 times and "cat" twice, so only 8 types are left: the, dog, chased, cat, and, ran, up, tree. The ratio is 8 ÷ 12, or 0.67. If you add a second sentence that reuses "the" and "cat", the tokens rise faster than the types, and the score drops.
The text length problem
TTR falls as a text gets longer, because the pool of common words runs out early. Every sentence needs "the", "of" and "is" again, but each one only counts as a new type the first time. Philip McCarthy and Scott Jarvis tested this in their 2010 validation study and found a correlation of r = .81 between text length and TTR. So a 200-word abstract will nearly always score higher than a 5,000-word report. That's why human writing has no single TTR. The figure shifts with length and genre. Compare texts of equal length, or use a standardized TTR that averages scores over fixed-size chunks.
Lexical diversity in AI text
AI-generated text scores lower on lexical diversity than human writing. Researchers at the Universidade da Coruña compared news articles from journalists with articles from language models. The human texts had a standardized TTR of 0.491, and every model they tested scored lower. The gap varies by model and by prompt, though. A low TTR score is a reason to look again.
Hapax ratio comparison
The hapax ratio counts the words that appear exactly once, divided by total tokens. TTR treats a word used once and a word used 40 times the same way, since each is still one type. The hapax ratio only rewards one-off words. In the example sentence, 6 words appear once, which gives a hapax ratio of 0.5. Read the 2 measures together because a text can keep a decent TTR while it leans on a small set of stock phrases.
How Snoika uses it
At Snoika, we score type-token ratio as 1 of 20 linguistic measures on every draft. It sits in the vocabulary group next to the hapax ratio and a count of repeated two- and three-word phrases. Our linguists set a human range for each measure. If a draft falls outside that range, we rewrite it before it goes to review. We pair TTR with the other 2 vocabulary measures because, as the length section shows, TTR on its own is easy to misread.
Related terms
These terms come up next to type-token ratio and measure vocabulary in different ways: