Weber-Wulff et al. 2023: testing of detection tools for AI-generated text

This record is one of eight in the studies index. The eight authors ran the Weber-Wulff 2023 tests themselves between March 7 and March 28, 2023, with Turnitin tested from April 14 to April 20 and PlagiarismCheck from May 2 to May 8 (journal version).

Weber-Wulff 2023 considered 18 tools and excluded 6, leaving the 14 below, tested in the March to May 2023 windows given above (arXiv 2306.15666). The paper names no product versions.

Detectors as listed in Weber-Wulff et al. 2023, DOI 10.1007/s40979-023-00146-z.

The sample held 54 documents in six classes of 9, written by nine researchers (the eight authors plus one collaborator), one document per class each. Each document went to each of the 14 tools, for 756 tests in Weber-Wulff 2023 (journal version).

Sample as described in Weber-Wulff et al. 2023, arXiv 2306.15666.

Each verdict was placed on a five-step scale from true positive to false negative (with mirror steps for human texts), and accuracy was computed four ways: binary, binary inclusive, semi-binary and logarithmic.

The table gives binary accuracy, false accusations over the 18 human documents, and undetected cases over the 36 AI documents, as Weber-Wulff 2023 reports them (arXiv 2306.15666, Tables 5, 9 and 10); tools without a card are shown where they top or close a table.

Values as printed in Weber-Wulff et al. 2023, DOI 10.1007/s40979-023-00146-z. The rows for Check For AI, Crossplag, PlagiarismCheck and Writeful GPT Detector, and the false accusation and undetected counts for every tool, were added on 9 November 2026 from Tables 5, 9 and 10 of the preprint (arXiv 2306.15666, read 9 November 2026), so that this record and the roster question, which lists all 14 tools with their status at that date, read from one source.

The logarithmic method in Weber-Wulff 2023 put Turnitin at 702 of 864 points (81%), Compilatio at 679 (79%) and Content at Scale last at 370 (43%) (arXiv 2306.15666). The authors' statement that all tools scored below 80% refers to accuracy averaged across the four methods in Weber-Wulff 2023; only 5 tools passed 70% on that average.

The translated class is read beside Chaka 2023's translation test on do AI detectors work on machine-translated text?. Human text was the easy case in Weber-Wulff 2023: overall accuracy on class 01-Hum was 96%, and accuracy dropped by 20% on the machine-translated class 02-MT (journal version). Edited AI text scored 42% in Weber-Wulff 2023 against 74% for unedited AI text, and machine-paraphrased text scored 26%.

Weber-Wulff 2023 estimates that approximately 20% of AI-generated texts would be misattributed to humans, rising to approximately 50% once the text is obfuscated (DOI 10.1007/s40979-023-00146-z). In Weber-Wulff 2023, 13 of the 14 tools missed at least one document in classes 03-AI and 04-AI; only Turnitin classified all of them correctly. False positive probability ranged from 0% (Turnitin) to 50% (GPT Zero) in Weber-Wulff 2023, and false negatives from 8% (GPT Zero) to 100% (Content at Scale).

The authors restrict their claims to English. Computer code was available but was left out of the tests, and the authors note that results from the same tool on the same text varied over time without a systematic replicability check. The repeatability question, added 9 November 2026, sets that note beside the five other records that address repeated runs or name a version, and the non-English evidence ranking counts this record's machine-translated class for nine cards.

The document set was also narrow by the authors' own account: no hybrid writing with iterative use of AI, the pattern they expect of students. Turnitin and PlagiarismCheck knew the documents were a test suite. Two authors help organise a conference sponsored by Turnitin and Compilatio, disclosed as a competing interest.

Twelve of the 14 detectors in Weber-Wulff 2023 have a card in this index (the Check For AI and Writefull cards, both tools since withdrawn, joined on 10 November 2026 with the tenth wave); the PlagiarismCheck card was added on 10 November 2026 with the ninth wave of cards, so the roster question now lists ten carded tools; the GPT-2 Output Detector card was added on 9 November 2026 and the Writer card later the same day, the detector having been withdrawn on 22 December 2025, and the Compilatio, DetectGPT and BrandWell cards (the last tested here as Content at Scale) were added with the fourth wave of cards the same evening. Each card quotes the result as reported here (DOI 10.1007/s40979-023-00146-z).

All twelve appear in the ranking of detectors by independent test record; the other two (Crossplag and the OpenAI classifier) have no card. How this record is sourced explains how study results enter a card, and the home page shows which AI detectors exist and who has tested them.

Related records: Perkins 2024 reused the five-step scale on adversarial edits; Walters 2023 tested 16 detectors on student essays months later (DOI 10.1007/s40979-023-00146-z).

Comparisons that read this record first: Winston AI vs Turnitin sets the two tools' rows from Tables 7, 9 and 10 side by side, since this is the one study that scored both (41 of 54 against 36 of 54, 0 false accusations in 18 for each), and ZeroGPT vs Turnitin does the same for that pair.