Liang et al. 2023: GPT detectors are biased against non-native English writers

This record is one of eight in the studies index. Liang 2023 is the record that put false positives against non-native writers on the agenda; the detectors were accessed on March 15, 2023, and the text generator was the March 14, 2023 version of ChatGPT 3.5 (arXiv 2304.02819). The preprint appeared on arXiv on April 6, 2023, with a third version on July 10, 2023.

Liang 2023 tested seven detectors, all accessed on the same day. The paper names no product versions beyond the access date.

Detectors as listed in Liang et al. 2023, DOI 10.1016/j.patter.2023.100779.

The human sample had two halves: essays by non-native writers and essays by native-speaking children. Liang 2023 added two bypass experiments and one perplexity analysis (arXiv 2304.02819v3).

Sample as described in Liang et al. 2023, arXiv 2304.02819.

The main metric is the false positive rate: the share of human essays each detector labelled AI-generated, averaged across the seven. The paper also counts essays flagged unanimously, the detection rate on generated text, and the perplexity of each text.

Liang 2023 reports the averaged figures below; per-detector false positive rates appear only as a chart in the paper and are left out of this record (DOI 10.1016/j.patter.2023.100779).

Values as reported in Liang et al. 2023, arXiv 2304.02819v3.

The headline is a gap between two human populations. Liang 2023 found near-perfect accuracy on the 88 US eighth-grade essays but misclassified over half of the 91 TOEFL essays, for an average false positive rate of 61.22% (DOI 10.1016/j.patter.2023.100779). In Liang 2023, 18 of the 91 TOEFL essays (19.78%) were called AI by all seven detectors at once, and 89 of 91 (97.80%) were flagged by at least one.

The second finding is that vocabulary moves the verdict. When Liang 2023 asked ChatGPT to enhance the word choices in the TOEFL essays to sound more like a native speaker, the average false positive rate fell from 61.22% to 11.77%, and only 1 of 91 essays (1.10%) was still flagged unanimously (arXiv 2304.02819). The reverse held too: simplifying the US eighth-grade essays in Liang 2023 pushed the misclassification rate from 5.19% to 56.65%.

The bypass experiments in Liang 2023 show the same lever working against detection. A second-round prompt asking ChatGPT to edit its own output cut detection on the 31 college essays from 100% to 13% in Liang 2023, and on the 145 scientific abstracts the detectors identified up to 68% before the self-edit and up to 28% after (arXiv 2304.02819v3). The authors tie both results to perplexity: detectors read low-perplexity, limited-vocabulary prose as machine output, whoever wrote it.

The authors call their sample sizes relatively small. They also note that most of the seven detectors in Liang 2023 used GPT-2 as the underlying model, so performance may differ with detectors built on GPT-3 or GPT-4 (DOI 10.1016/j.patter.2023.100779).

The Liang 2023 analysis covers perplexity-based and supervised-learning methods that were popular in early 2023, which the authors say may not represent all detection techniques. One inconsistency sits in the primary text itself: the Discussion refers to a GPT-4 intervention while the Methods name ChatGPT 3.5. No corrigendum is recorded (DOI 10.1016/j.patter.2023.100779).

Seven detectors, one day in March 2023, and 91 essays from a single forum in Liang 2023: the bias is documented for that set, and any claim about today's tools needs today's test (DOI 10.1016/j.patter.2023.100779).

Six of the seven detectors in Liang 2023 have a card in this index; the sixth, AI Writing Check, the Quill.org and CommonLit tool the paper lists as "Quil.org (aiwritingcheck.org)", was added on 10 November 2026 with the eleventh wave for a service its vendor took down in August 2023, and carries the averaged 61.22% TOEFL figure as its one record; the fifth, the GPT-2 Output Detector, is the tool the paper lists by its Hugging Face address and was added on 9 November 2026. Each card carries the averaged false positive figure with the base stated, because the paper gives no per-detector number in its text.

These cards also appear in the ranking of detectors by independent test record. How this record is sourced sets out why averaged figures are attributed to the study rather than to any single card, and the home page lists which AI detectors exist and who has tested them.

Related records: Weber-Wulff 2023 found the same pattern on machine-translated human text, and Perkins 2024 tested "write as a non-native English speaker" as a deliberate evasion technique.