Perkins et al. 2024: AI detectors against adversarial techniques

This record is one of eight in the studies index. The March 2024 preprint carried a different title (arXiv 2403.19148); the headline figures are identical in both versions. Perkins 2024 measures evasion directly: every manipulation was made by the generative tool that wrote the text, none by hand.

Perkins 2024 lists seven detectors and reports results for all seven, although the abstract and methods say "six"; the inconsistency is the paper's own.

Detectors as listed in Perkins et al. 2024, DOI 10.1186/s41239-024-00487-w.

The sample started from 15 AI texts of about 500 words each. Perkins 2024 then produced 89 altered versions with six adversarial techniques (one case failed, so 89 rather than 90) and added 10 human-written controls, for 114 samples and 805 tests, of which 797 were valid (arXiv PDF).

Sample as described in Perkins et al. 2024, journal version.

Perkins 2024 reused the five-step classification from Weber-Wulff 2023 and computed accuracy by the binary, semi-binary and logarithmic methods plus their mean. False accusations (FAS) and undetected cases (UDC) count false and partially false verdicts at a 60% threshold (DOI 10.1186/s41239-024-00487-w).

The table gives baseline mean accuracy (Table 7 of the paper), before-and-after figures under manipulation (Table 8) and error counts (Table 10) as printed in Perkins 2024, for the detectors with a card plus the top and bottom of each table (arXiv PDF).

Values as printed in Perkins et al. 2024, Tables 7, 8 and 10, DOI 10.1186/s41239-024-00487-w. Table 8 baseline figures differ from Table 7 for the same tools; both are as the paper prints them.

Perkins 2024 reports low numbers before any manipulation: mean accuracy of 39.5% on the original AI samples and 67% on the human controls across seven tools (DOI 10.1186/s41239-024-00487-w). Copyleaks detected 64.8% of AI texts in Perkins 2024, followed by Turnitin at 61%, with GPTZero at approximately 26%.

The generator mattered in Perkins 2024: Bard output was identified 76.9% of the time, GPT-4 23.9% and Claude 2 17.7% (journal version).

Manipulation cut accuracy to a mean of 22.1% across the six techniques in Perkins 2024, a drop of 17.5% (arXiv PDF). Spelling errors were the strongest technique in Perkins 2024 at 12.9% accuracy (a 27% drop), then burstiness at 15.9% (24% drop), paraphrase at 18.4% (21% drop), lower complexity at 21% (19% drop) and non-native voice at 27.7% (12% drop); higher complexity barely moved at 37% (2% drop). After manipulation, Copyleaks held the highest accuracy in Perkins 2024 at 58.7% and GPTKit the lowest at 4.5%, while Turnitin showed the largest drop, 42.1%.

The authors restrict the findings to a relatively small number of samples, three generative tools and seven detectors, which they say do not cover the range of writing styles, manipulations or tools, and to a specific snapshot in time.

The spelling-error samples, the strongest evasion, are ones the authors say a student would be very unlikely to submit. Eight samples failed to process on some tools in Perkins 2024, which is why 797 of 805 tests were valid. No corrigendum is recorded (DOI 10.1186/s41239-024-00487-w).

Six of the seven detectors in Perkins 2024 have a card in this index; the GPTKit card, added on 10 November 2026 with the ninth wave, carries its 6% mean accuracy on unmanipulated AI text falling to 4.5% after adversarial techniques, the lowest post-manipulation figure of the seven, and 0 false accusations in 10 human samples. Each card quotes each quoting the baseline, the post-manipulation figure and the error counts with their bases; the GPT-2 Output Detector card, added on 9 November 2026, carries its 57.2% baseline, the fall from 34.7% to 17.5%, 0 false accusations in 10 and 81 of 104 undetected.

All six appear in the ranking of detectors by independent test record. Since 9 November 2026 the post-manipulation figures feed the robustness ranking, where Copyleaks' 58.7% is the highest Perkins figure and GPTKit's 4.5% the lowest of the seven (GPTKit's ranked value is the higher 30% from Orenstrakh 2023 under the highest-printed rule, and the 4.5% stays in its base), and the undetected counts feed the false negative rate ranking. How this record is sourced explains how a before-and-after result is stored on a card, and the home page lists which AI detectors exist and who has tested them.

Related records: Weber-Wulff 2023 supplied the classification scale; the RAID benchmark tests eleven attacks at scale.