Do AI detectors give the same result twice? What the studies measured, what the vendors say about model changes, and what nobody has measured
None of the 35 study records in the index measured whether an AI detector returns the same score when the same text is submitted twice, and the one record that mentions the question, Weber-Wulff 2023, states that results from the same tool on the same text "varied over time" without a systematic replicability check (DOI 10.1007/s40979-023-00146-z, read 8 November 2026). Six records tested each text once and say so or describe their test as a snapshot in time, and 9 of the 72 cards carry a dated model change on the vendor's own pages, from Turnitin's seven release notes between 24 May 2023 and 14 October 2026 to Pangram's two model cards of 29 July and 19 October 2026, by the vendor pages read 8 and 9 November 2026 in the data file (Turnitin release notes, read 8 November 2026; Pangram 4.1 model card, read 9 November 2026). The question therefore has two halves with different answers: the same text in the same minute has not been tested, and the same text months apart meets a different model.
Replicability means two things for a detector. The first is run-to-run: the same text, the same tool, the same day, two scores. The second is version-to-version: the same text scored against a model the vendor has since replaced. The first table lists what each study record states about repeated runs and about the version it tested; the second lists every dated model change a vendor publishes on a card in the index, because a score printed in a 2023 paper and a score returned today come from different software. This page is one of the questions answered from the index; its siblings ask whether a 100% accurate AI detector exists and which detectors claim a false positive rate under 1%, and the AI detector register carries every card. The ranking by independent testing counts the records below per card.
| Study record | Repeated runs | Version or date of the tool tested | Source | Read date |
|---|---|---|---|---|
| Weber-Wulff 2023 | Each of 54 documents sent once to each of 14 tools, 756 tests; the authors note that results from the same tool on the same text varied over time, with no systematic replicability check | No product versions named; tests run 7 to 28 March 2023, Turnitin 14 to 20 April, PlagiarismCheck 2 to 8 May 2023 | DOI 10.1007/s40979-023-00146-z | 8 November 2026 |
| Perkins 2024 | 805 tests on 114 samples, 797 valid, one pass per sample; the authors limit the findings to "a specific snapshot in time" | Free versions of GPTZero, ZeroGPT (retrieved 3 March 2024) and the GPT-2 Output Detector; purchased Copyleaks and GPTKit credits; an institutional Turnitin licence | DOI 10.1186/s41239-024-00487-w | 8 November 2026 |
| Kar 2024 | One run per tool on each of four texts on 12 January 2024; "no threshold, repeat runs or statistics beyond percentages are reported" | Free tiers of ten tools, no versions named | DOI 10.1177/02537176241247934 | 9 November 2026 |
| Van Vlasselaer 2026 | 160 papers scored once by each of four tools; findings "valid for a snapshot in time"; preregistered on AsPredicted (#228383) | Fully AI papers generated May 2025; no detector versions named | DOI 10.1007/s40979-026-00226-w | 9 November 2026 |
| Jabarian and Imas 2025 | 1,992 human and 7,968 AI passages scored once per detector; the authors "do not expect the results to remain constant over time" and recommend routine published audits | Pangram API v2025-05, GPTZero API v2, Originality.ai Turbo endpoint, RoBERTa-base | NBER w34223 | 9 November 2026 |
| Epoch AI 2026 | 495 human and 594 AI passages scored once at each detector's default verdict in June and July 2026; versions "can be updated by their makers without notice" | Pangram 3.3.2, GPTZero model 2026-05-11-base, Originality.ai Turbo 3.0.2 | epoch.ai | 9 November 2026 |
Statements as each record quotes them from the primary source on the date shown. The other 24 records neither repeat a run nor discuss the question, so the table does not list them; none of the 35 reports a second score on the same text.
Has any study submitted the same text twice?
No record in the index reports it. Weber-Wulff 2023 comes closest and says so in its limits: the eight authors ran 756 tests between March and May 2023, one per document per tool, and observed that the same tool gave different results on the same text at different times, a variation they recorded as an observation rather than a measurement (DOI 10.1007/s40979-023-00146-z, read 8 November 2026; Weber-Wulff 2023 record). Kar 2024 is explicit that it reports no repeat runs on its four texts (DOI 10.1177/02537176241247934, read 9 November 2026; Kar 2024 record). Perkins 2024 and Van Vlasselaer 2026 each describe their results as a snapshot in time (DOI 10.1186/s41239-024-00487-w, read 8 November 2026; DOI 10.1007/s40979-026-00226-w, read 9 November 2026).
The nearest thing to a second run is a vendor rerunning a study's texts on a newer model. Pangram's 4.1 model card of 19 October 2026 lists 0.000% false positives and false negatives on the 15,936 full-length texts of Jabarian and Imas 2025, where the published study recorded 0.0000 false positives for Pangram at threshold 0.5 (Pangram 4.1 model card, read 9 November 2026; vendor benchmarks record). That is the same texts scored by a different model version, run by the vendor, so it answers the version-to-version question for one tool and one dataset and says nothing about run-to-run noise.
Which vendors say the model behind the score has changed?
Nine cards carry a dated change, and a score from before the date was produced by different software.
| Detector | Dated model changes on the vendor's pages | Source | Read date |
|---|---|---|---|
| Turnitin | Minimum raised from 150 to 300 words, 24 May 2023; paraphrase detection, 6 December 2023; asterisk on scores of 1% to 19%, 16 July 2024; bypasser-tool detection, 27 August 2025; recall updates, 14 October 2025 and 12 February 2026; English model update, 14 October 2026; none of the notes carries a number | Turnitin release notes | 8 November 2026 |
| GPTZero | Model 4.3b on the benchmarking page of 5 February 2026, updated 20 July 2026; model 2026-05-11-base scored by Epoch AI in June and July 2026; GPTZero 4o announced 18 October 2026 with a "revamped" architecture; an October 2023 model scored in Pangram's 2024 report | GPTZero benchmarking | 9 November 2026 |
| Pangram | Pangram 4 model card, 29 July 2026, 41 false positives in 1,000,000 texts; Pangram 4.1 model card, 19 October 2026, 70 in 781,981 on a cleaned set; 3.2 scored by GPTZero and 3.3.2 by Epoch AI; the headline rate moved because the test set changed as well as the model | Pangram 4.1 model card | 9 November 2026 |
| Copyleaks | Model V11 tested 10 September 2026 and published 28 September 2026 on the methodology page | copyleaks.com | 8 November 2026 |
| Sapling | Model upgrade of 17 September 2026: human-text false positives fell by 60% on the vendor's modern-text set, no absolute rate published | sapling.ai | 8 November 2026 |
| Originality.ai | Model 1.4 scored in Walters 2023; 3.0.2 Turbo (1.5% false positive rate, vendor figure) scored by Epoch AI in 2026; 5.0 and 5.0 Max launched with the post read on 8 November 2026 and an accuracy post of 25 October 2026 | originality.ai | 8 November 2026 |
| Codequiry | "Our current model, v3e, is trained on Claude Opus 5.5, GPT-5.6 Sol and GPT-5.5 Codex submissions" | codequiry.com | 9 November 2026 |
| Illuminarty | Release post of 18 March 2023: binary AI detection validation accuracy 91.8% on the previous model and 97.5% on the new one, "within our collected dataset" | illuminarty.ai | 9 November 2026 |
| Codeleaks | Release notes of 16 December 2025: the AI verdict no longer returned for source code files | docs.copyleaks.com | 9 November 2026 |
Changes as each vendor dates them on the page linked, read on the date shown; a vendor that publishes no dated change is absent from the table, which says nothing about whether its model changed.
Turnitin's list is the longest and the least numeric: seven dated changes to the English model and its display since the April 2023 launch, none with a before-and-after figure (Turnitin release notes, read 8 November 2026; Turnitin claims record). The three studies that scored Turnitin, Weber-Wulff 2023, Walters 2023 and Perkins 2024, all did so before 6 December 2023, so every Turnitin figure in the index predates every change in the table (Walters 2023 record). Pangram's two cards show how a vendor's own headline moves between versions: 41 in 1,000,000 on 29 July 2026 and 70 in 781,981 on 19 October 2026, a change the card attributes to cleaning the test set as well as to the model (Pangram 4 model card, read 9 November 2026).
A dated leaderboard entry is one way to fix a result in time: the RAID entries for 6 cards carry a submission date each, from GLTR's 10 June 2019 to Winston AI's 24 September 2026, as the leaderboard question, read 9 November 2026, records.
Why would the same text get a different score?
Two reasons are on the record and one is not. The first is a model change, dated in the table above: a paper's score and today's score are two models' answers. The second is the threshold: Jabarian and Imas 2025 reports Pangram's false positive rate as 0.0000 at a fixed 0.5 threshold and at most 0.0075 per genre at Youden-optimised thresholds, and GPTZero's as 0.0071 at every fixed threshold, so the same model on the same texts yields different error rates depending on where the line is drawn (NBER w34223, read 9 November 2026). Turnitin's FAQ adds a third source of movement in its own words: in documents of only a few hundred words "the prediction will be mostly all or nothing", so a small edit can move a short document from one end to the other (guides.turnitin.com, read 8 November 2026). What is not on the record is run-to-run noise with the model and threshold held fixed: no vendor page in the index states whether its classifier is deterministic, and no study has tested it.
A detector score carries the date it was produced as part of its meaning, and a score without a date and a model version cannot be compared with another.
Questions about AI detector consistency
Do AI detectors give different results for the same text?
Over time, yes, by the one observation on the record: Weber-Wulff 2023 notes that results from the same tool on the same text varied over time, without a systematic check (DOI 10.1007/s40979-023-00146-z, read 8 November 2026). In the same minute, no one has measured it: none of the 35 indexed records reports a repeated run.
Has any study tested AI detector replicability?
No: 0 of the 35 records submitted a text twice to the same tool; six describe one pass per text or a snapshot in time, and Kar 2024 states that no repeat runs are reported (DOI 10.1177/02537176241247934, read 9 November 2026). Jabarian and Imas 2025 recommends routine published audits for that reason (NBER w34223, read 9 November 2026).
How often does Turnitin change its AI detection model?
Seven dated changes between 24 May 2023 and 14 October 2026 on Turnitin's release notes page, including recall updates on 14 October 2025 and 12 February 2026 and an English model update on 14 October 2026, none with a published before-and-after figure (Turnitin release notes, read 8 November 2026).
Does a study result still apply to today's version of a detector?
Not as a measurement of today's model: every Turnitin figure in the index predates the first dated model change of 6 December 2023, and Epoch AI 2026 pinned its versions in June 2026 and notes they "can be updated by their makers without notice" (epoch.ai, read 9 November 2026). A study result is a dated fact about a dated model.
Why did Pangram's published false positive rate change in 2026?
Because the test set changed with the model: 41 in 1,000,000 FineWeb texts on the Pangram 4 card of 29 July 2026 became 70 in 781,981 on the 4.1 card of 19 October 2026 after the vendor removed "spam, advertising and programmatically generated documents" from the set (Pangram 4.1 model card, read 9 November 2026).
Sources
All read 9 November 2026 unless an earlier date is shown.
- Weber-Wulff et al. 2023: https://doi.org/10.1007/s40979-023-00146-z (read 8 November 2026)
- Perkins et al. 2024: https://doi.org/10.1186/s41239-024-00487-w (read 8 November 2026)
- Kar et al. 2024: https://doi.org/10.1177/02537176241247934
- Van Vlasselaer et al. 2026: https://doi.org/10.1007/s40979-026-00226-w
- Jabarian and Imas 2025, NBER w34223: https://www.nber.org/papers/w34223
- Epoch AI 2026, AI detectors rarely flag human writing: https://epoch.ai/data-insights/ai-detectors-false-negatives
- Turnitin AI writing detection model release notes: https://guides.turnitin.com/hc/en-us/articles/28294949544717-AI-writing-detection-model (read 8 November 2026)
- Turnitin AI writing detection FAQs: https://guides.turnitin.com/hc/en-us/articles/28477544839821-Turnitin-s-AI-writing-detection-capabilities-FAQs (read 8 November 2026)
- GPTZero AI detection benchmarking, 5 February 2026, updated 20 July 2026: https://gptzero.me/news/gptzero-ai-detection-benchmarking-the-industry-standard-in-accuracy-transparency-and-fairness/
- Pangram 4.1 model card, 19 October 2026: https://www.pangram.com/research/model-card/pangram-4-1
- Pangram 4 model card, 29 July 2026: https://www.pangram.com/research/model-card/pangram-4
- Copyleaks testing methodology: https://copyleaks.com/ai-detector/testing-methodology (read 8 November 2026)
- Sapling AI detector: https://sapling.ai/ai-content-detector (read 8 November 2026)
- Originality.ai 5.0 launch post: https://originality.ai/blog/introducing-originality-5-0-and-max (read 8 November 2026)
- Codequiry AI code detection: https://codequiry.com/detect-ai-written-code
- Illuminarty release post, 18 March 2023: https://illuminarty.ai/en/posts/big-spring-update.html
- Copyleaks release notes: https://docs.copyleaks.com/resources/updates/release-notes
Revision log: 2026-11-09: page created from the study records and the dated vendor changes on 42 cards at index version 2026.11.7.; 2026-11-09: eighth-wave pages linked
Related pages: Which AI detector vendors publish their test data? Every benchmark page, model card and methodology article in the index, with what can be downloaded, Which AI detectors run on-premise or offline? The four that run on your own machine, the one open-source text component, the one on-device PC tool and the ten vendors that state an on-premise option, AI detectors ranked by independent testing Two records added in the fifth wave on 9 November 2026, Pegoraro 2023 and Fleckenstein 2024, changed nine counts and none: Pegoraro 2023 tested ZeroGPT, GPTZero, Copyleaks, Originality.ai, Writer, Content at Scale (BrandWell), DetectGPT, the GPT-2 Output Detector and GLTR, lifting GPTZero to 12, the GPT-2 Output Detector to 10, Copyleaks to 7 and Originality.ai and ZeroGPT to 6; Fleckenstein 2024 tested 289 teachers and no software, so it counts for no card ([Fleckenstein 2024 record](/studies/fleckenstein-2024-teachers/)). Illuminarty, one of four cards added in the same wave, joined with its one image record from Ha and Passananti 2024, so 18 of 72 cards now qualify. and Questions about AI detectors, answered from the index.