SAFE 2025: twelve teams' synthetic voice detectors on 21 real sources, 13 TTS models and laundered audio

The record is the second audio entry in the studies index, after ASVspoof 5: here participants never saw any of the data, and the top five teams are named (arXiv PDF, Sections 3.2 and 5, read 9 November 2026).

The detectors are research submissions, not products. Round 1 ran for about 70 days from March 2025 and drew more than 700 submissions from 12 teams; every submission ran on the organisers' hardware (one NVIDIA T4 GPU, 10,000-second limit) with no internet access, and had to output a binary "real" or "generated" decision per clip at a threshold of the team's choosing. Table 5 names the five teams ranked on Task 1 (arXiv PDF, Sections 3.2 and 5 and Table 5, read 9 November 2026).

Teams as printed in Table 5 of arXiv 2510.03387; the other seven Round 1 teams are not named.

None of the voice cards in this index, Pindrop Pulse, Resemble Detect, Reality Defender, Hive, Sensity or AI or Not, is named as a participant, baseline or sponsor (arXiv PDF, Table 5 and Acknowledgments, read 9 November 2026).

The real audio is 200 clips from each of 21 sources, 4,200 clips and 18.25 hours averaging 15.64 seconds, kept in original codec and sampling rate, from Mandarin, English, Arabic, German, Japanese and Russian speech across podcasts, radio, phone calls, audiobooks, cassettes and video-platform audio. The generated audio is 200 clips from each of 13 text-to-speech models, 2,600 clips averaging 12.24 seconds: Cartesia, ElevenLabs, Fish, HierSpeech, Kokoro, Parler, StyleTTS, Edge, F5, MetaVoice, OpenAI, SeamlessM4T (Meta, listed in the paper by its family name) and Zonos. Ten real sources and seven generators sat in a public split whose anonymised scores participants could see; the private split, used for the final ranking, held everything (arXiv PDF, Section 4 and Tables 1 and 2, read 9 November 2026).

Tasks as described in Sections 3.1 and 4.2 and Tables 3 and 4; the real clips were unchanged in every task. The abstract gives 17 TTS models and 21,000 samples where the body gives 13 models and 22,700; the body says 18 augmentations and Table 3 lists 19 rows including the unaugmented clips. This record carries the body and table figures.

The metric is balanced accuracy, the mean of the true positive rate and the true negative rate, on the private split at each team's own decision threshold (arXiv PDF, Section 3.3, read 9 November 2026).

Every figure below is printed in Tables 6 to 10 of the paper (arXiv PDF, read 9 November 2026).

Values as printed in arXiv 2510.03387, Tables 6 to 10. Per-generator, per-source and per-condition balanced accuracies combine the conditional rate with the team's Table 6 rate on the other class.

ISP led all three tasks, but ANO rose from fourth to second after compression (0.74 to 0.78), and the authors note that post-processing "actually improved detector performance overall" for ANO and JAI (arXiv PDF, Section 5.1, read 9 November 2026). DMF's 0.67 in Task 1 came from a threshold that called half the real clips fake (TNR 0.49).

Newer generators were harder: Zonos (0.71) and Edge (0.68) against ElevenLabs (0.97) and SeamlessM4T (0.93) for ISP; rare-language real audio produced the most false alarms. Neural codecs barely hurt, which the authors attribute to their similarity to the decoders inside the generators, while Gaussian noise, Opus, phone audio and chained codecs each cut ISP by 0.16 to 0.20. Playing a clip through a speaker and recording it again cut ISP from 0.87 to 0.62 and VIP from 0.77 to 0.50, "a potential and easily exploitable vulnerability" (arXiv PDF, Section 5.1, read 9 November 2026).

The paper calls its findings "initial insights" from Round 1, with Round 2 not reported; participants received only "top-level accuracy across a subset of anonymized sources"; and performance on public-split generators was only slightly higher than on private ones, which the authors read as either real generalisation or training sets that already contained those models (arXiv PDF, Sections 1, 3 and 5.1, read 9 November 2026). Evident from the paper: the abstract and body disagree on the model count (17 against 13) and sample count (21,000 against 22,700); every figure is at a team-chosen threshold, so TPR and TNR trade off differently per team; and per-condition tables rest on 20 or 50 clips per model.

No card takes a result line, because no indexed product participated. The record counts toward the voice column of studies by modality as a record with zero indexed detectors, and the content type view lists the eleven cards that state voice. No figure enters the false positive rate ranking or the adversarial robustness ranking, which rank cards, though the laundering drop is the audio counterpart of the paraphrase drops those rankings carry for text. The home page lists which AI detectors exist and who has tested them.

Related records: ASVspoof 5 scored anonymised teams on 680,774 read-speech utterances with cost-based metrics; RAID 2024 measures text detectors under adversarial edits as Task 3 measures audio detectors under laundering.

The challenge names neither of the two voice vendors the Pindrop vs Resemble Detect comparison sets side by side, and scores no music system; the music detectors question lists the five music cards with no record, both read 10 November 2026.