Gao et al. 2023: ChatGPT-generated scientific abstracts against the GPT-2 Output Detector and blinded reviewers

This record joins the studies index as the only record built on scientific abstracts and the only one to put a detector beside human readers on the same texts. The paper was received on 1 January 2023 and accepted on 30 March 2023, and is open access (nature.com article page, read 9 November 2026).

Gao 2023 used one AI output detector, the GPT-2 Output Detector, a RoBERTa-based classifier that returns a "fake" score from 0.02% to 99.98%. Two similarity checkers, the free Plagiarism Detector website and the paid iThenticate, and four blinded human reviewers were run on the same abstracts for comparison; neither checker is an AI detector, so neither has a row below (DOI 10.1038/s41746-023-00819-6, Methods, read 9 November 2026).

Detector as named in Gao et al. 2023, DOI 10.1038/s41746-023-00819-6.

The 50 original abstracts came from issues of Nature Medicine, JAMA, NEJM, BMJ and The Lancet published in late November and December 2022. For each, ChatGPT (the 15 December version) was given the prompt "Please write a scientific abstract for the article [title] in the style of [journal] at [link]" in a new session, and the first output was kept without editing (DOI 10.1038/s41746-023-00819-6, Methods, read 9 November 2026).

Sample as described in Gao et al. 2023, DOI 10.1038/s41746-023-00819-6.

The detector's scores were compared between the two sets and summarised as an ROC curve. The reviewers, four study team members used to reading abstracts, were told the set was a mixture, given 25 abstracts each and asked for a binary original-or-generated verdict with free-text notes; for each pair of reviewers an electronic coin flip decided which of them saw the original and which the generated version (DOI 10.1038/s41746-023-00819-6, Methods, read 9 November 2026).

Gao 2023 reports medians with interquartile ranges for the scores and a single ROC curve for the detector; the reviewer figures are percentages of the abstracts in each class (DOI 10.1038/s41746-023-00819-6, Results and Figure 2, read 9 November 2026).

Values as printed in Gao et al. 2023, DOI 10.1038/s41746-023-00819-6.

The interquartile range is the detail to read: the lower quartile of the generated abstracts scored 12.73% "fake", so a quarter of ChatGPT's abstracts sat at or below that score while the real abstracts clustered at 0.02%, and Gao 2023 reports an optimal cutoff of 1.46% that gave sensitivity of 86% and specificity of 94% (DOI 10.1038/s41746-023-00819-6, Results, read 9 November 2026).

The reviewers and the detector disagreed about which abstracts were hard. Gao 2023 reports that the detector's scores did not differ (p = 0.45) between generated abstracts the reviewers caught and those they missed, and that reviewers described suspected abstracts as vague and formulaic, sometimes doubting real ones for including clinical trial registration numbers or alternative spellings (DOI 10.1038/s41746-023-00819-6, Results and Figure 4, read 9 November 2026).

The similarity checkers point the other way: generated abstracts scored a median 0% on the Plagiarism Detector website and 27 on iThenticate against 62.5% and 100 for the originals, because the text was new even though the numbers in it were invented; the patient cohort sizes ChatGPT invented correlated with the real ones at r = 0.76 (DOI 10.1038/s41746-023-00819-6, Results and Figure 1, read 9 November 2026). The authors conclude that AI output detectors "may serve as an editorial tool to help maintain scientific standards" depending on a publisher's guidelines.

The authors name the small sample size and the few reviewers first. They used a single fixed prompt without exploring variations, took only ChatGPT's first output for each title, and note that the model gives a different response each time, so the study "likely underestimates the ability of ChatGPT to generate scientific abstracts" (DOI 10.1038/s41746-023-00819-6, Discussion, read 9 November 2026).

The GPT-2 Output Detector accepts at most 510 tokens, so some abstracts could not be fully evaluated. The reviewers knew that part of their set was generated, which an editor reading a submission would not, and gave a binary verdict with no formal rubric (DOI 10.1038/s41746-023-00819-6, Discussion, read 9 November 2026). The paper tested no other AI detector and no field outside biomedical science, both of which the authors leave to future work.

An AUROC of 0.94 in Gao 2023 describes how well one 2019 detector separated 50 December 2022 ChatGPT abstracts from 50 journal abstracts (DOI 10.1038/s41746-023-00819-6); it is not an accuracy rate for any product on any other text.

The one detector in Gao 2023 has a card in this index. The card records the AUROC of 0.94 on 100 abstracts, the 86% sensitivity and 94% specificity at the 1.46% cutoff, and the medians of 99.98% and 0.02% (DOI 10.1038/s41746-023-00819-6, read 9 November 2026).

The card appears in the ranking of detectors by independent test record, and the 94% specificity on 50 real abstracts is the kind of figure the false positive rate ranking draws on. How this record is sourced explains how a specificity on a stated base enters a card, and the home page lists which AI detectors exist and who has tested them.

Related records: Weber-Wulff 2023 tested the same GPT-2 Output Detector on 54 documents that spring and scored it 39 of 54, and Orenstrakh 2023 found it tied second of eight tools on 134 English coursework texts with 2 false positives in 114 (DOI 10.1038/s41746-023-00819-6, read 9 November 2026).

The researchers view reads this record as the index's only result on scientific writing, beside the nineteen cards whose stated limits or upload formats take a full manuscript.