How AI detectors work, from perplexity to calibrated thresholds

Most AI detectors work in one of two ways. Some are classifiers trained on labelled examples of human and AI writing. Others are zero-shot methods that ask a language model how predictable a text is. Either way the output is a score, and the decision that matters is where to draw the line. This explainer covers how AI detectors work, why the threshold matters as much as the model, and exactly what ours does with your text.

How AI detectors work: two main approaches

A detector never sees who wrote a text. All it has is the text itself, and it looks for statistical patterns that showed up more often in AI writing than in human writing when it was built or tested. The two main families find those patterns differently.

Zero-shot detectors use an existing language model as a measuring instrument. They need no examples of AI text to learn from. Trained classifiers learn from thousands of examples labelled "human" or "AI" and work out their own cues.

Zero-shot detectors: how predictable is this text?

A language model assigns a probability to every possible next word. After "The cat sat on the", the word "mat" gets a high probability and "tax return" a very low one. Perplexity summarises how surprised the model is, on average, by the words a text actually uses. Because language models generate text by picking likely words, AI writing tends to have low perplexity.

Researchers have built on that idea in several steps:

• GLTR (2019) colour-coded each word by how highly a model ranked it. In a study with human participants, the visual hints raised people's success at spotting generated text from 54% to 72%.
• DetectGPT (2023) compares a passage with slightly reworded versions of itself. Text a model generated tends to sit at a local peak of that model's probability, so the rewordings score lower. On fake news generated by the GPT-NeoX model, it raised AUROC from 0.81 for the best earlier zero-shot method to 0.95.
• Fast-DetectGPT (2024) replaced the expensive rewording step with sampling. Its authors report about a 75% relative improvement over DetectGPT and a speed-up of roughly 340 times.
• Binoculars (2024) compares how two closely related language models see the same text. Its authors report that it caught over 90% of samples from ChatGPT and other models at a false positive rate of 0.01%, without being trained on ChatGPT data.

The appeal of zero-shot methods is that they don't need retraining for every new model. The weakness is that some human writing is predictable too. A 2023 Stanford study traced detectors' high false positive rates on essays by English learners to exactly this: low perplexity (we cover it in AI detectors and non-native English). Results also vary a lot outside the original papers. In an ACL 2025 study of 300 non-fiction articles, Binoculars caught 40.0% of the AI articles at its strict setting, and on long fiction in the StoryScope study it scored 55.9% macro-F1, far below trained classifiers.

Trained classifiers: learning from labelled examples

A classifier usually starts from a pretrained language model such as RoBERTa and is fine-tuned on a large set of texts labelled human or AI. It isn't told what to look for. It learns whatever separates the two piles: word choice, grammar, rhythm, structure, often in combinations no person would spot.

On text like its training data, a good classifier can be extremely accurate. In StoryScope, a ModernBERT classifier trained on the study's stories scored 99.9% macro-F1 on held-out stories from the same collection. In a working paper for the University of Chicago's Becker Friedman Institute, the best-performing commercial detector, Pangram, had near-zero error rates on medium and long passages, while an open-source RoBERTa baseline was judged unsuitable for high-stakes use.

The weakness is generalisation. Across the 11 generators in the RAID benchmark (ACL 2024), detectors lost accuracy on models and sampling settings they hadn't been trained on, and simple adversarial attacks fooled them easily. A classifier trained mostly on one genre can fail on others: the US Federal Trade Commission alleged that a detector sold as 98% accurate had been trained only on academic content, and testing found 53% accuracy on general-purpose text.

Why thresholds and calibration matter

Whatever the method, a detector produces a score. Turning that score into "AI" or "human" means choosing a threshold, and the threshold sets the balance between catching AI text and wrongly flagging people. Move it down and the detector catches more, and accuses more. Move it up and both fall.

Small rates add up. A threshold that flags 1 human text in 100 sounds strict, but across 10,000 human essays it means about 100 people wrongly flagged. That is why a responsible threshold starts from the false positive rate you can accept, measured on human writing that was kept apart from the data used to build the detector. Jabarian and Imas propose a version of this in that working paper: decide how many errors you can tolerate, set that as a cap, and compare detectors at the cap.

Calibration is the other half. A calibrated score is one whose meaning has been checked against real data, so that a given number corresponds to something measured rather than to an arbitrary scale. Length matters too: a 60-word paragraph carries much less evidence than a 2,000-word essay, so a single threshold for all lengths will be wrong for some of them.

How IsThisAiWritten works

Our detector is a trained classifier with calibrated thresholds, and all of it runs in your browser.

1. The model downloads once. It is TMR, an open RoBERTa-based classifier of 131 MB, fetched from Hugging Face and cached by your browser. Your text is never uploaded.
2. The text is cleaned. Invisible characters such as zero-width spaces, and look-alike letters, are found and can be removed. The text is checked for language (English only) and split into sentences.
3. Windows are scored. The text is cut into overlapping windows and the classifier scores each one. That is why there is no length limit.
4. Scores become passages. A smoother joins the window scores into passages, and a calibrated combiner turns them into a probability for the whole text and an AI fraction, the share of words in AI-like passages.
5. Fixed rules decide. Cut-offs are set per length band so that at most about 5% of our human calibration texts are flagged. The outcome is Human, Mixed, AI or Inconclusive, with a confidence word. Under 50 words there is no result, and confidence stays low until about 150 words.
6. The report explains. You get an AI-likeness figure ("more AI-like than X% of human writing we tested"), a sentence map that is relative within your text, and style signals that are explanations only and play no part in the verdict. Fiction also gets a descriptive narrative fingerprint.

Our open benchmark sets out how this setup is tested on a frozen set of human and AI texts. Like any detector, it can be wrong in both directions, so treat a result as a reason to look closer, not as proof.

What no detector can tell you

Every detector, whatever its method, is answering a narrow statistical question: does this text resemble the AI writing it was built or tested on? It cannot tell you who wrote the text, or whether AI was used to brainstorm, to fix grammar or to write the whole thing. It can't know how a model released after its last test will write, which we discuss in can AI detectors catch the newest models. A detector score is one piece of evidence, to be weighed with drafts, version history and a conversation with the writer.

Sources

1. Gehrmann, Strobelt and Rush, GLTR: Statistical Detection and Visualization of Generated Text (ACL 2019 demo), checked October 2026
2. Mitchell et al., DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (ICML 2023), checked October 2026
3. Bao et al., Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature (ICLR 2024), checked October 2026
4. Hans et al., Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text (2024), checked October 2026
5. Dugan et al., RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors (ACL 2024), checked October 2026
6. Liang et al., GPT detectors are biased against non-native English writers (Patterns, 2023), checked October 2026
7. Russell, Karpinska and Iyyer, People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text (ACL 2025), checked October 2026
8. Russell, Rajendhran, Pham, Iyyer and Wieting, StoryScope: Investigating idiosyncrasies in AI fiction (COLM 2026), checked October 2026
9. Jabarian and Imas, Artificial Writing and Automated Detection, BFI Working Paper 2025-116, checked October 2026
10. FTC: FTC Order Requires Workado to Back Up Artificial Intelligence Detection Claims (28 April 2025), checked October 2026