Chaka 2023: detecting AI content in ChatGPT, YouChat and Chatsonic responses with five tools

This record is one of nine in the studies index. Chaka 2023 is the only study indexed here that tested detectors on machine-translated text, and the only one whose sample contains no human-written texts at all. The paper was received on 24 June 2023, accepted on 22 July 2023 and carries a CC BY licence (journal page, read 9 November 2026).

Chaka 2023 tested five detectors, all in their free or non-premium versions, between 30 March and 2 April 2023. The paper names no product versions, so the table carries the detector as named and its card (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026).

Detectors as named in Chaka 2023, DOI 10.37074/jalt.2023.6.2.12.

The texts came from three chatbots answering English prompts on applied English language studies, the author's own research field: six ChatGPT prompts on 31 January 2023, seven YouChat prompts and eight Chatsonic prompts on 7 March 2023, each prompt producing one response. The six ChatGPT responses were then run through Google Translate into German, French, Spanish, Southern Sotho and isiZulu and fed to GPTZero, the one detector the author says recognised all five languages at the time; the German, French and Spanish translations were also fed to Copyleaks, which did not support the two African languages (Wayback copy of the PDF, read 9 November 2026).

Sample as described in Chaka 2023, DOI 10.37074/jalt.2023.6.2.12.

Each response was pasted unedited into each detector and the output recorded as correct (AI-generated), misclassified (human-written) or indecisive. The paper describes an exploratory design and states that none of the texts was edited, paraphrased or otherwise manipulated before detection (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026).

Chaka 2023 reports the detection outcomes as counts in the text of its Results section and in Tables 1 to 3, which are images in the PDF. The counts below are taken from the Results text; the overall rank is the paper's own (Wayback copy of the PDF, read 9 November 2026).

Counts from the Results section of Chaka 2023, DOI 10.37074/jalt.2023.6.2.12.

The pattern differed by chatbot. On the six ChatGPT responses in Chaka 2023, GPTZero and the OpenAI classifier each classified 4 as AI-generated and were indecisive on 2, Copyleaks classified 5 correctly and 1 as human, while Writer and GLTR each classified 5 of the 6 as human-written (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026). On the seven YouChat responses, GPTZero classified 1 correctly and 6 as human, Copyleaks 5 correctly and 2 as human, and GLTR classified all 7 as human-written. On the eight Chatsonic responses, Copyleaks classified 5 correctly, GPTZero 3 with 4 misclassified and 1 indecisive, and the OpenAI classifier was indecisive on 5 of the 8.

The translation test is the paper's distinctive result, and the question page on machine-translated text sets it beside the Weber-Wulff 2023 result on translated human text. Chaka 2023 reports that GPTZero classified every one of the 30 machine-translated ChatGPT responses as human-written, and attributes the outcome to the high perplexity scores the translated texts received: the highest were 2,478 for German, 1,715 for Southern Sotho and 938 for isiZulu, against a range of 38 to 90 on the English ChatGPT responses (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026). Copyleaks, tested on three of the languages, classified 14 of 18 translated texts as AI-generated, with its weakest result on German at 3 of 6.

The author's conclusion in Chaka 2023 is that all five tools "seem not yet fully ready" to detect machine-generated text across contexts, and that the misclassifications show a tendency toward false negatives, the term Weber-Wulff et al. 2023 call misattributing AI text to humans (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026).

The author names the use of free or non-premium versions as the major limitation of Chaka 2023, noting that all five tools had only free versions when the data were collected in March and April 2023 and that most have since added paid tiers (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026). The prompts all came from one subject area and one author, and the three sets of responses number 6, 7 and 8.

The translation test in Chaka 2023 measures whether GPTZero and Copyleaks recognised Google-translated AI text as machine-generated, and the author states that the accuracy of the translations themselves was not at issue. The Southern Sotho output used the Lesotho orthography, and Google Translate labels isiZulu as Zulu (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026).

Four of the five detectors in Chaka 2023 have a card in this index; the Writer card was added on 9 November 2026 for a detector withdrawn on 22 December 2025. Each card records the English counts on a base of 21 responses and, for GPTZero and Copyleaks, the translation counts on bases of 30 and 18 (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026).

All four appear in the ranking of detectors by independent test record. The paper also names the GPT-2 Output Detector, Originality.ai and Crossplag in a list of other tools, none of which entered the test. How this record is sourced explains why counts read from a results section are shown with the section named. The home page lists which AI detectors exist and who has tested them.

Related records: Weber-Wulff 2023 tested GPTZero and the OpenAI classifier on 54 documents in the same spring and also tested machine-translated human text, and Elkhatat 2023 tested four of the same five tools, with human controls, on 35 paragraphs (DOI 10.37074/jalt.2023.6.2.12, read 9 November 2026).