Fleckenstein et al. 2024: do teachers spot AI? 89 novice and 200 experienced teachers on ChatGPT and student essays

This record joins the studies index as its human baseline: no software detector was tested, and the raters were teachers, the group most often asked to decide whether a student used ChatGPT. The authors are at the University of Hildesheim, the Leibniz Institute for Science and Mathematics Education in Kiel, the Zurich University of Teacher Education and Kiel University; the article is CC BY 4.0 (DOI 10.1016/j.caeai.2024.100209, title page, read 9 November 2026).

Fleckenstein 2024 tested no detection software. Two groups of human raters judged whether each of four essays was written by ChatGPT or by a student, so the table lists the rater groups (DOI 10.1016/j.caeai.2024.100209, Sections 4.1.1 and 5.1.1, read 9 November 2026).

Rater groups as described in Fleckenstein et al. 2024, DOI 10.1016/j.caeai.2024.100209, Sections 4.1.1 and 5.1.1. Study 1 raters were aged 21 to 38; Study 2 raters were aged 22 to 77, mostly in the UK and North America.

Every rater saw four argumentative essays answering one TOEFL iBT writing prompt on whether a teacher's ability to relate to students matters more than subject knowledge. The student essays came from the MEWS corpus of upper-secondary students in Germany and Switzerland, written in 30 minutes in English as a foreign language and scored 2 (low) or 4 (high) on the 0 to 5 TOEFL rubric. The two ChatGPT essays were generated by prompting ChatGPT with the rubric descriptors for level 2 and level 4. Study 1 used two preselected student essays; Study 2 drew a low-quality and a high-quality student essay at random for each rater and reused the same two ChatGPT essays (DOI 10.1016/j.caeai.2024.100209, Sections 4.1.2 and 5.1.2, read 9 November 2026).

Sample as described in Fleckenstein et al. 2024, DOI 10.1016/j.caeai.2024.100209, Sections 4.1.2, 4.2.1, 5.1.2 and 5.2.1.

Raters answered yes or no to "This text was written by an artificial intelligence", rated confidence from "50%, I guessed" to "100%, very sure", then scored language, structure, content and overall quality from 1 to 7. Study 1 raters were told that half the texts were AI-generated; Study 2 raters only that some were (DOI 10.1016/j.caeai.2024.100209, Sections 4.1.3 and 5.1.3, read 9 November 2026).

Tables 1 and 4 of Fleckenstein 2024 give the share of essays whose source was identified correctly; the prose gives confidence means and tests (DOI 10.1016/j.caeai.2024.100209, Sections 4.2 and 5.2, read 9 November 2026).

Values as printed in Fleckenstein et al. 2024, DOI 10.1016/j.caeai.2024.100209, Tables 1 and 4 and Sections 4.2.1, 4.2.2, 5.2.1 and 5.2.2.

In Study 1 the pre-service teachers' judgments were unrelated to the real source: the chi-square test gave p = .825, which the authors read as raters who "cannot correctly identify the source of the texts", at either quality level (DOI 10.1016/j.caeai.2024.100209, Section 4.2.1, read 9 November 2026). Only the assumed source predicted confidence: raters who thought an essay was AI-written were less sure (B = minus 9.68, p < .01).

In Study 2 the experienced teachers called 83.0% of low-quality student essays and 89.0% of low-quality ChatGPT essays student-written (p = .084). On high-quality essays they were right on 63.0% of student essays and 64.5% of ChatGPT essays (p < .001). The authors conclude that experienced teachers were "better able to correctly identify the source of high-quality texts" but called weak essays student work (DOI 10.1016/j.caeai.2024.100209, Section 5.2.1, read 9 November 2026).

On quality scores, pre-service teachers gave the lowest marks to student essays they correctly took for student work, and experienced teachers scored high-quality ChatGPT essays higher than high-quality student essays (DOI 10.1016/j.caeai.2024.100209, Sections 4.2.3 and 5.2.3, read 9 November 2026).

The authors list three limits of Study 1: the raters were novices who may lack the training and language proficiency for the task, they were told that half the texts were AI-generated, and only two student essays were used. Study 2 changed the population, the disclosure and the text selection at once, so the authors "do not know for sure what change has caused divergent findings", and they recommend that future studies not disclose the share of AI texts (DOI 10.1016/j.caeai.2024.100209, Sections 4.3 and 6.1, read 9 November 2026). Both studies reused the same two ChatGPT essays, and all essays answered one TOEFL prompt. The discussion adds that detection software "does not seem to solve this issue either", citing a report that the OpenAI classifier flagged many student texts.

What this record does not say: the 45.1% and 37.8% figures are human judgments of two ChatGPT essays by raters told that AI texts were present, so they are not a false negative rate for any software detector and do not measure how teachers perform on unannounced AI use.

No detector card in this index was tested by Fleckenstein 2024. The record's value to the index is the human baseline: teachers identified 45.1% and 37.8% of ChatGPT essays, figures that software results on the false negative rate ranking and the false positive rate ranking can be read against (DOI 10.1016/j.caeai.2024.100209, Tables 1 and 4, read 9 November 2026). The ranking of detectors by test record counts software records only. The article's open-access licence, CC BY 4.0, and its publication dates are confirmed in the publisher's Crossref deposit (api.crossref.org, read 9 November 2026), and the researchers view reads the record beside Gao 2023's blinded reviewers. The home page lists which AI detectors exist and who has tested them.

Related records: Gao 2023 is the other human-reviewer record, where blinded reviewers caught 68% of ChatGPT abstracts and flagged 14% of real ones; the OpenAI classifier record carries the tool the authors cite; Weber-Wulff 2023 tested 14 software tools on 54 documents in the same period (DOI 10.1016/j.caeai.2024.100209, Section 6.2, read 9 November 2026).