Orenstrakh et al. 2023: detecting LLM-generated text in computing education, eight detectors on 164 submissions

This record joins the studies index as the only record whose human texts are real student assignments: the submissions date from 2016 to 2018, before ChatGPT existed, so none can have been written with it (arXiv 2307.07411, Section 3, read 9 November 2026).

Orenstrakh 2023 tested eight publicly available detectors in April 2023, chosen from January to February 2023 and limited to tools that appeared to offer their own models; apparent copies of the open-source GPT-2 detector were excluded. Six have a card in this index, the GPTKit card added on 10 November 2026 (arXiv 2307.07411, Section 3.1 and Table 2, read 9 November 2026).

Detectors as named in Orenstrakh et al. 2023, arXiv 2307.07411, Table 2.

The 124 human submissions came from two publicly funded research-focused institutions, one in North America and one in South America: 101 reports from a third-year databases course and 13 exam answers from a networking course, both in English, and 10 final thesis project papers in Spanish. ChatGPT produced 30 submissions from the same assignment prompts, 20 in English and 10 in Spanish, and 10 of the English ChatGPT submissions were run through QuillBot. Texts under 1,000 characters were excluded and texts over 2,500 characters were cut at the last full sentence, giving 1,312 predictions across the eight tools (arXiv 2307.07411, Section 3 and Table 1, read 9 November 2026).

Sample as described in Orenstrakh et al. 2023, arXiv 2307.07411.

Each detector's output was converted to a probability of AI authorship, using the tool's own number where it gave one and a conversion table for category labels; GLTR, which gives no overall score, was wrapped in a z-score and sigmoid calibrated on 20 human and 20 ChatGPT texts. Accuracy was counted by a threshold method, where a text is correct if its probability falls on the right side of 50%, and by a weighted average (arXiv 2307.07411, Sections 3.1 and 3.2, read 9 November 2026).

The figures below are from Tables 6, 8, 9 and 12 of Orenstrakh 2023, with the threshold method used for accuracy; the human base for the English tables is 114 submissions and the ChatGPT base is 20 (arXiv 2307.07411, read 9 November 2026).

Values as printed in Orenstrakh et al. 2023, arXiv 2307.07411, Tables 6, 8, 9 and 12.

Combining the two English columns, Orenstrakh 2023 puts CopyLeaks first at 97.06%, the GPT-2 Output Detector and CheckForAI at 96.62%, GLTR at 88.73%, GPTKit at 87.50%, the OpenAI detector at 77.37% and GPTZero at 49.69% (arXiv 2307.07411, Section 4.1, read 9 November 2026). The paper's conclusion names CopyLeaks the most accurate detector, GPTKit the best at avoiding false positives and GLTR the most resilient to paraphrasing; GPTKit's own row reads 100.00% on human text and 75.00% on ChatGPT text.

The 52 GPTZero false positives are the figure the authors single out: Orenstrakh 2023 states that GPTZero "highlighted 52 human submissions as either possibly or entirely AI-generated" and that the sentence highlights gave no clear reason for the flag (arXiv 2307.07411, Section 4.2, read 9 November 2026). On the weighted-average method in Table 7, GPTZero scored 73.95% on human text and 55.00% on ChatGPT text.

On the Spanish texts, Orenstrakh 2023 reports that CopyLeaks and the AI Text Classifier called every submission AI-generated, human or not, while the GPT-2 Output Detector, GPTZero, CheckForAI and GLTR called every ChatGPT submission human; averaging Tables 11 and 12, ChatGPT accuracy fell to about 17.50% (arXiv 2307.07411, Sections 4.1 and 5, read 9 November 2026). After QuillBot, average ChatGPT accuracy across Tables 9 and 10 was about 49.17%. The Spanish result places this record on the non-English evidence ranking, added 9 November 2026, for four cards, and the QuillBot paraphrase result places it among the eleven records on the paraphrased text question, added the same day.

The authors list four threats to validity: the results reflect the detectors as they stood in April 2023 and could change with any model update; accuracy, false positives and resilience may not exhaust what effectiveness means; the data came from two institutions, one working in English and one in Spanish; and more data sets would strengthen the findings (arXiv 2307.07411, Section 6, read 9 November 2026). The Spanish and paraphrase sets hold 10 texts each.

The authors mined over 2,000 submissions from programming, data structures and compilers courses but could not use them, because removing the code left too little context; every text tested in Orenstrakh 2023 is prose (arXiv 2307.07411, Section 3, read 9 November 2026). The GLTR probability is the authors' own construction on top of a tool that reports none.

A false positive count in Orenstrakh 2023 is a score past the halfway mark on the authors' conversion scale for a pre-ChatGPT assignment from one of two institutions, measured in April 2023; it is not a rate for any other text, institution or date.

Six of the eight detectors in Orenstrakh 2023 have a card in this index; the GPTKit card joined on 10 November 2026 with the ninth wave of cards. Each card records the Table 6 accuracy pair on bases of 114 human and 20 ChatGPT English submissions, the Table 8 false positive count in 114, and the Table 9 figure after QuillBot on 10 texts (arXiv 2307.07411, read 9 November 2026).

All six appear in the ranking of detectors by independent test record, and the false positive counts in 114 are the kind of figure the false positive rate ranking draws on: GPTKit's 0 of 114 is its ranked rate there, and its 30% after QuillBot is its value in the robustness ranking, both entered on 10 November 2026. QuillBot was the paraphraser here, not a detector under test; its card covers the separate QuillBot AI detector. How this record is sourced explains how a count enters a card, and the home page lists which AI detectors exist and who has tested them.

Related records: Weber-Wulff 2023 tested GPTZero, the GPT-2 Output Detector and the OpenAI classifier on 54 documents in the same spring, and Walters 2023 ran Copyleaks, GPTZero and Originality on 126 essays that summer (arXiv 2307.07411, read 9 November 2026).