RAID: the shared benchmark for AI text detectors

This record is one of eight in the studies index. RAID differs from the other seven: L. Dugan and co-authors built a dataset and a public leaderboard rather than a one-off test, with data and code on GitHub (arXiv 2405.07940). This record reports only figures printed in the paper.

RAID 2024 evaluated 12 detectors, 8 open and 4 commercial, with no product versions stated (DOI 10.18653/v1/2024.acl-long.674).

Detectors as listed in Dugan et al. 2024, DOI 10.18653/v1/2024.acl-long.674.

RAID 2024 spans 11 generators, 8 domains, 11 adversarial attacks and 4 decoding strategies; 10% of the dataset is held back without labels as a hidden test set, and RAID-extra adds 2.3 million generations of Python code, Czech news and German news (arXiv 2405.07940).

Sample as described in Dugan et al. 2024, ACL Anthology.

The main metric in RAID 2024 is accuracy (true positive rate) at a fixed false positive rate of 5%, with the threshold chosen per detector to hit that rate. RAID 2024 also reports false positive rates at naive thresholds of 0.25, 0.5, 0.75 and 0.95 (DOI 10.18653/v1/2024.acl-long.674).

The table gives false positive rates at a naive threshold (Table 4 of the paper), selected accuracy figures at 5% false positive rate (Table 5) and accuracy under selected attacks (Table 6), as printed in RAID 2024 (arXiv 2405.07940).

Values as printed in Dugan et al. 2024, Tables 4 to 6, DOI 10.18653/v1/2024.acl-long.674. ZeroGPT could not reach the target false positive rate in some Table 5 cells, which the paper marks with an asterisk.

The first finding in RAID 2024 concerns calibration. Open-source detectors, especially metric-based ones, showed dangerously high false positive rates at naive thresholds, with GLTR at 99.3% and LLMDet at 96.0% at a threshold of 0.5, while no closed-source detector exceeded 1.7% at any threshold in RAID 2024 (arXiv 2405.07940).

The second concerns decoding. RAID 2024 reports that a repetition penalty decreased accuracy by up to 32 points regardless of decoding strategy (ACL Anthology). Table 5 of RAID 2024 shows GPTZero at 98.8 on open chat models under greedy decoding and 9.4 on open non-chat models under sampling.

The third concerns attacks. Metric-based methods such as Binoculars degraded by as much as 36.1% when a few words were swapped for synonyms in RAID 2024, and every detector was sensitive to homoglyphs except GPTZero, which lost 0.3% while five others dropped an average of 40.6% (DOI 10.18653/v1/2024.acl-long.674). Originality fell from 85.0 to 9.3 under homoglyphs in RAID 2024 yet rose to 96.7 under paraphrase.

The authors conclude that detectors are not yet ready for widespread deployment or high-stakes use: many were nearly inoperable at low false positive rates, failed to generalise across decoding strategies, showed bias toward certain models and domains, and degraded under simple attacks. They single out Binoculars at very low false positive rates, Originality for precision in constrained scenarios, and GPTZero for holding up under attack. The ethics statement opposes any disciplinary or punitive use of detectors.

The authors write that no dataset can ever fully cover every way a detector might be fooled, and that RAID lacks multilingual text across many domains. As language models improve, a fixed dataset of generated text goes stale.

The hidden test labels are kept private, and the public leaderboard is a live resource whose values change; any figure quoted from the leaderboard needs the date read. The published test data question, added 9 November 2026, uses RAID as its non-vendor reference row, with data and code public and the hidden labels private, and the false positive studies question counts this record's human-document rates for six cards.

A detector tuned to a 5% false positive rate on the RAID 2024 dataset has that rate on this dataset; the same threshold on a classroom's essays can land anywhere (DOI 10.18653/v1/2024.acl-long.674).

All four commercial detectors in RAID 2024 have a card in this index, each recording the Table 4 false positive rate, the Table 5 range and the homoglyph result where printed, and two open detectors gained cards on 9 November 2026: the GPT-2 Output Detector, which RAID 2024 ran as RoBERTa (GPT-2) in base and large sizes, and GLTR; a third, Fast-DetectGPT, run as F-DetectGPT with the repository defaults, gained a card on 10 November 2026 and records the Table 4 rates (23.2% at threshold 0.5), the Table 5 range (98.6 to 0.6), 73.6 with no attack and 71.8 under paraphrase, 34.0 under synonym swap and 51.4 under homoglyphs at 5% FPR (Table 16) (DOI 10.18653/v1/2024.acl-long.674). A fourth open detector, RADAR, queried as TrustSafeAI/RADAR-Vicuna-7B, gained a card on 10 November 2026 with the eleventh wave and records the Table 4 rates (3.48% at threshold 0.5), the Table 5 range (88.8 to 31.8), 70.9 with no attack and 67.3 under paraphrase, 59.3 under homoglyphs and 82.2 under zero-width spaces at 5% FPR (Tables 6 and 16), and the 44.7 total on the non-English and code domains (Table 7).

All nine appear in the ranking of detectors by independent test record. How this record is sourced explains why leaderboard values stay off cards until read on a stated date, and the home page shows which AI detectors exist and who has tested them.

Related records: Perkins 2024 tested six adversarial techniques on 114 samples, and Weber-Wulff 2023 tested paraphrase and human editing on 54 documents (DOI 10.18653/v1/2024.acl-long.674).

Comparisons that read this record first: Winston AI vs Originality.ai sets the two tools' Table 4 and Table 5 cells side by side, since RAID is the one record that scored both (0.55% against 0.25% false positives at threshold 0.5, 97.2% against 98.6% at a fixed 5% false positive rate on open-source chat models), and both vendors cite the benchmark on their own pages.