Rashidi et al. 2023: 14,400 pre-ChatGPT journal abstracts through the RoBERTa GPT-2 detector

The record joins the studies index as the largest false-positive test of a single detector on scientific writing. The paper was received on 28 July 2023, accepted on 10 October 2023, and is open access under a CC BY-NC-ND licence (DOI 10.1016/j.jpi.2023.100342, read 10 November 2026).

Rashidi 2023 used one detector: the "roberta-base-openai-detector" model from the OpenAI repository on Hugging Face, downloaded with the transformers Python package and run locally to return a 0 to 1.0 probability that a text is AI-generated. The authors state that no GPT-3 or GPT-4 based detection API was available to them and that cost limited them to the freely available model (DOI 10.1016/j.jpi.2023.100342, Methods, read 10 November 2026). The same weights sit behind the GPT-2 Output Detector demo tested in Gao 2023.

Detector as named in Rashidi et al. 2023, DOI 10.1016/j.jpi.2023.100342, Methods.

The abstracts were scraped from PubMed, from the first five pages of search results for each journal, and sampled in nine five-year clusters from 1980 to 1984 through 2020 to February 2023, 1,600 drawn at random per cluster and 14,400 in all, with a length filter so that the abstracts fit the model's token limit (DOI 10.1016/j.jpi.2023.100342, Methods, read 10 November 2026).

Sample as described in Rashidi et al. 2023, DOI 10.1016/j.jpi.2023.100342.

A 50% probability was the base threshold for a suspected AI call, with further cuts at 60%, 70%, 80% and 90%; full-length abstracts were used for the main analysis, and the dilution study fed the model the first half and first quarter of each abstract as a positive control (DOI 10.1016/j.jpi.2023.100342, Methods, read 10 November 2026).

Rashidi 2023 prints its results as shares of each 1,600-abstract range (Fig. 2, a bar chart with a table beneath) and of each journal's abstracts (Table 1), with the maxima stated in the Discussion (DOI 10.1016/j.jpi.2023.100342, Results and Discussion, read 10 November 2026).

Values as printed in Rashidi et al. 2023, DOI 10.1016/j.jpi.2023.100342, Fig. 2, Table 1, Results and Discussion, read 10 November 2026.

The trend over time is the finding the authors spend most of the Discussion on. The share of real abstracts scored above 90% peaks in 2000 to 2004 at 5.13% and declines steadily to 0.75% in 2020 to 2023, which the authors attribute in part to temporal misalignment between the detector's Reddit-derived training text and scholarly prose written decades earlier, while noting that this does not "fully explain all the observed trends" (DOI 10.1016/j.jpi.2023.100342, Discussion, read 10 November 2026).

The journal effect was larger than the time effect. Four journals showed shares above 90% between 1.89% and 3.89%, while NEJM "was shown to be an outlier at 10.24% for unknown reasons"; the authors tested abstract length and abstract format as causes and found that neither separates NEJM from the other four journals (DOI 10.1016/j.jpi.2023.100342, Discussion and Appendix Fig. 1, read 10 November 2026). The supplementary material shows one real 1982 NEJM abstract that the detector scored at 99.96% AI.

The dilution study found the detector "performed best on the full abstract" and worse on the first half and first quarter (Fig. 3, no figures printed in the text), and the ten flagged abstracts re-run through a GPT-4 prompt in the ChatGPT Plus interface were all called human, but so were a separate set of AI-written abstracts, so the authors call that check "rather non-specific" and "not a validated method" (DOI 10.1016/j.jpi.2023.100342, Discussion and Supplementary Table 1, read 10 November 2026).

The authors name the single detector first: the study "only evaluates a single AI-text detection model, which is not the most current technology available", chosen because no newer detector offered API access for large-scale analysis and because of the expense of commercial tools (DOI 10.1016/j.jpi.2023.100342, Discussion, read 10 November 2026). The 2020 to 2023 figure carries its own caveat: the assumption that no abstract could have used a language model no longer holds for that range, so "the true value of the false detection rate is less than or equal to the quoted value of 0.75%".

Evident from the paper: no AI-written abstracts were scored by the detector, so the study reports no miss rate and no accuracy; the figures appear per range and per journal rather than as one pooled false positive rate; and the length filter applied at collection is not stated as a number.

The one detector in Rashidi 2023 has a card in this index. The card records the maximum of 8.69% of 1,600 real abstracts above 50% and 5.13% above 90%, the 0.75% for 2020 to 2023 and the NEJM share of 10.24% above 90% (DOI 10.1016/j.jpi.2023.100342, read 10 November 2026).

The card appears in the ranking of detectors by test record, and the per-range shares above 50% are the kind of figure the false positive rate ranking draws on. How this record is sourced explains how a share on a stated base enters a card, and the home page lists which AI detectors exist and who has tested them.

Related records: Gao 2023 ran the hosted demo of the same model on 50 ChatGPT and 50 real abstracts from five of the same journals and reports 94% specificity at a 1.46% cutoff; Jabarian and Imas 2025 ran the same RoBERTa-base weights on 1,992 human passages and reports a false positive rate of 0.9503 at a 0.5 threshold (DOI 10.1016/j.jpi.2023.100342, read 10 November 2026).

The record is one of five listed with its figure and base in the Q3 2027 studies digest, and the one of the five that scores an indexed card, read 10 November 2026.