Sadasivan et al. 2023: can AI-generated text be reliably detected? Recursive paraphrasing against four detector classes

This record joins the studies index as a stress test rather than a product comparison: Sadasivan 2023 attacked four classes of research detector with a paraphraser and a spoofing method, and proved a bound on how well any detector can do (arXiv 2303.11156, read 9 November 2026).

Sadasivan 2023 tested four classes of detector: watermarking, trained neural network classifiers, zero-shot statistical detectors and a retrieval-based detector; the question page on how AI detectors work sorts every card in the index into those classes. One of them, OpenAI's RoBERTa GPT-2 detector in its base and large sizes, has a card in this index; the rest are research methods with no product behind them (arXiv 2303.11156v4, Sections 2.3, 2.4 and Appendix A, read 9 November 2026).

Detectors as named in Sadasivan et al. 2023, arXiv 2303.11156.

The text came from the XSum news dataset: the first 300 tokens of 1,000 articles prompted a watermarked OPT-1.3B or OPT-13B model to produce 1,000 AI passages of about 300 tokens, and the next 300 tokens of each article served as the 1,000 human passages. The trained and zero-shot detectors were tested on 1,000 passages generated by GPT-2 Medium from the same prompts (arXiv 2303.11156v4, Sections 2.3 and 2.4, read 9 November 2026).

Sample as described in Sadasivan et al. 2023, arXiv 2303.11156.

The attack rewrote each AI passage with a paraphraser, then fed the paraphrase back in for up to five rounds (pp1 to pp5), using DIPPER with 11 billion parameters, LLaMA-2-7B-Chat, or a 222-million-parameter T5 model that works one sentence at a time. Detection was scored as the true positive rate at a 1% false positive rate (TPR at 1% FPR) and as AUROC (arXiv 2303.11156v4, Section 2.1, read 9 November 2026).

The headline figures below are the ones Sadasivan 2023 states in its introduction; the per-section figures that follow are from Sections 2.3, 2.4 and 3 and from Table 5 of the appendix, with the base named for each (arXiv 2303.11156v4, read 9 November 2026).

Values as printed in Sadasivan et al. 2023, arXiv 2303.11156.

Two rounds of paraphrasing were enough in every watermark setting: Sadasivan 2023 states that pp2 brought TPR at 1% FPR below 50% in all settings and pp5 below 20%, and that watermarked texts of 50 or 100 tokens were detected no better than chance after the attack (arXiv 2303.11156v4, Section 2.3 and Figure 4, read 9 November 2026). With ten paraphrases and ten queries to the detector, the RoBERTa-Large figure fell to 0.792 in Table 5; the paper calls the trained RoBERTa detectors the most resistant class and notes that the MAGE training set includes XSum passages.

In the MTurk study of Sadasivan 2023, 70% of DIPPER paraphrases were rated high quality for content preservation and 89% for grammar, on 20 watermarked passages over five rounds, each rated by three workers (arXiv 2303.11156v4, Section 2.2 and Table 3, read 9 November 2026).

Theorem 1 of Sadasivan 2023 bounds the AUROC of any detector at one half plus the total variation distance between the human and AI text distributions minus half that distance squared, so a distance below 0.2 caps the best possible detector below 0.7 (arXiv 2303.11156v4, Section 4, read 9 November 2026). The spoofing section reports that the retrieval detector could be made to flag a human essay 100% of the time by asking the model to paraphrase it first.

The paper's human study covers 20 passages per paraphraser, and the authors write that the tolerable quality loss is application-specific (arXiv 2303.11156v4, Section 2.2, read 9 November 2026). The authors state that the trained detectors may have been trained on the same human XSum text used here, and that out-of-distribution human text, which they did not test, is a separate failure mode.

Watermark detection held up better on longer passages in Sadasivan 2023, and the authors say whether watermarking is reliable for long texts needs more investigation. Estimating the total variation distance between real text distributions from finite samples is, in the authors' words, a challenging problem (arXiv 2303.11156v4, Section 4, read 9 November 2026).

Two of the detectors in Sadasivan 2023 have a card in this index; the DetectGPT card was added on 9 November 2026 with the fourth wave and records the AUROC fall from 96.5% to 59.8% after one T5 paraphrase round on 1,000 GPT-2 Medium passages and to 25.2% under recursive paraphrasing. The GPT-2 Output Detector card records the RoBERTa-Large and RoBERTa-Base figures from Table 5 on a base of 1,000 GPT-2 Medium passages, and the introduction's 100% to 60% statement (arXiv 2303.11156v4, read 9 November 2026).

Both cards appear in the ranking of detectors by independent test record, and DetectGPT's post-paraphrase figure from Krishna 2023 places it in the adversarial robustness ranking. The watermarking, threshold and retrieval methods have no card because none is a product a reader can run on a document; DetectGPT received one because its MIT-licensed code is a runnable release and three indexed studies tested it. How this record is sourced explains why introduction and section figures are both shown when they differ, and the home page lists which AI detectors exist and who has tested them.

Related records: Krishna 2023 built the DIPPER paraphraser used here and proposed the retrieval detector that Sadasivan 2023 broke; the RAID benchmark later ran paraphrase attacks on commercial detectors (arXiv 2303.11156, read 9 November 2026).