How AI Detectors Actually Work: Perplexity and Burstiness Explained

Every AI detector, from the free ones to the ones your university pays for, is trying to answer the same question: does this text look like it came from a predictable pattern-matcher, or a person? The two oldest ideas for answering that are perplexity and burstiness. They're not the whole story anymore, but they're still the reason AI writing gets flagged in the first place, and understanding them explains a lot about why detectors get it wrong as often as they get it right.

What is perplexity?

Perplexity measures how surprised a language model is by the words in front of it. Feed a sentence through a model and ask it to predict each next word: if the actual word was the model's top guess almost every time, perplexity is low. If the writing keeps picking words the model didn't expect, perplexity is high. Low perplexity means predictable, safe word choices. High perplexity means the writing takes turns a statistical model wouldn't have guessed.

Low perplexity happens almost by definition. To be a language model, one picks the statistically probable next word for the output, word after word—that is the whole machine. Thus when a detection tool is reporting low perplexity, it is usually reporting precisely the reason language models actually are language models: conservative, expected word-choices, word-choice after word-choice. There is no hard-and-fast rule, but a commonly reported threshold is perplexity of 85, with text above being human-like and text below seeming machine-written.

sbb-itb-1768945

What is burstiness?

Using that same measurement, perplexity, burstiness is how that perplexity varies from sentence to sentence within a given document. Human writing is never perfectly smooth. Some sentences are simple and direct, some ramble, some meander for days before sharply taking a turn no one could have seen. This variation, which is the standard deviation of the perplexity at the sentence level, is burstiness. Because artificial language models are designed to be coherent and maintain a predictable cadence, there is a steady rhythm of sentence structures and a constant, often linear rate of predictability over the course of a piece. Low burstiness, smooth and predictable, is clearly artificial. High burstiness, some sentences mundane and others surprising, is likely human.

Add these two together and you have four rough areas. A low perplexity with low burstiness (predictable wording, nothing surprising) would be the most telltale sign of AI. A high perplexity with a high burstiness (random words, wild variation) a sure sign of human writing. In the middle ground, where so much writing exists, detectors become uncertain.

Where the method breaks down

And that's why it falls apart. Perplexity isn't about detecting if a human wrote a text; it's about measuring how well a particular language model was able to predict it. Run the Declaration of Independence through a perplexity checker and it's possible the output looks quite artificial — because that exact phrasing occurs so frequently in the training data models were assembled from, and models are all about reducing perplexity on that kind of text. The exact same issue applies to Wikipedia articles. Perplexity isn't finding AI. It's finding commonness.

Perplexity varies with the model. You could say that a word choice is predictable for model X but not for Y, and then each subsequent model release changes the definition of predictability. A detector trained to identify tokens that "last year's GPT used to predict" doesn't necessarily "identify what this year's GPT predicts," and when a commercial model is closed source, there is no guarantee the provider exposed its token probabilities at the word level, most don't.

The most consequential failure mode is what it does to non-native English writers. A 2023 Stanford study run against 91 human-written TOEFL essays found that seven widely-used GPT detectors misclassified them as AI-generated at an average rate of 61.22%. Almost 98% of those essays got flagged by at least one detector. The reason is structural, not incidental: non-native writers tend to use simpler sentence structures and a narrower vocabulary, which produces exactly the low-perplexity, low-burstiness signal the detectors are built to catch, whether or not any AI was involved.

What detectors use now

Detector companies know this, which is why the leading tools have mostly moved on. GPTZero has said publicly that it stopped relying on perplexity and burstiness as its primary method back in 2023, shifting to a deep-learning classifier trained directly on examples of AI and human writing, with perplexity kept around only as one signal among several rather than the main one. Modern detectors generally work more like a spam filter than a formula: a model trained on huge sets of labeled human and AI text learns to recognize deeper patterns, word frequency distributions, sentence and paragraph structure, punctuation habits, that go well beyond a single predictability score, and in principle can keep improving as it sees more data.

What this means for your writing

None of this says detectors are bunk, or that none of these are worth looking at. But it says one number taken out of context is low-hanging fruit, not grounds for indictment — particularly for people who are non-native English speakers, who write in a literal style, or who are writing about something that lends itself to predictable formulations. If your work shows up consistently with the same boring rhythm and cadence to the computer, don't get bad, get varied. That, after all, is what burstiness is really trying to catch.

Frequently asked questions

Do AI detectors still use perplexity and burstiness?

Some do, but usually as one signal among several rather than the main method. GPTZero, one of the most widely used detectors, has said it moved to a deep-learning classifier as its primary approach back in 2023, with the older statistical metrics kept in as supporting signals.

Why do AI detectors flag writing by non-native English speakers?

Because non-native writers tend to use simpler sentence structures and a narrower vocabulary, which produces the same low-perplexity, low-burstiness pattern detectors associate with AI text. A 2023 Stanford study found detectors misclassified real TOEFL essays as AI-generated 61.22% of the time on average.

Can a low perplexity score alone prove text is AI-generated?

No. Perplexity measures how predictable text is to one specific language model, not who wrote it. Text that closely matches common phrasing, like well-known historical documents or Wikipedia-style writing, can score as low-perplexity even when it's entirely human.

What's the difference between perplexity and burstiness?

Perplexity measures how predictable individual word choices are. Burstiness measures how much that predictability varies from sentence to sentence across an entire piece. Low values in both are the strongest signal detectors associate with AI writing.

---

• Sources used for this draft (for your own reference, not necessarily to publish):
• gptzero.me — What is perplexity & burstiness for AI detection?
• pangram.com — Why Perplexity and Burstiness Fail to Detect AI
• arxiv.org — GPT detectors are biased against non-native English writers (Liang et al., Stanford, 2023)