A Corrected Protocol, a Benchmark, and What Patching Buys
Ali Alavi · Donald S. WilliamsonDepartment of Computer Science and Engineering, The Ohio State University
Speech quality predictors such as NISQA, DNSMOS and UTMOS are now used as training rewards, so it
matters how much score an optimiser can extract without improving the audio. We fix two flaws in how that is
measured, benchmark four widely used predictors under one protocol, and audit whether patching a predictor on
its own hacks makes it a safer reward.
Utterances hacked by the same attacker
One black-box attacker per predictor in the EnCodec latent space, perturbation size ε = 0.5,
500 BVCC test utterances, five attackers each. A hack raises the score more than τ = 0.3 MOS above the
codec round trip while an independent panel says the audio got worse.
NISQA89.6%NISQA: hacked on 89.6% of utterances (mean of 5 attackers), 87.6% to 91.6% across attackers; 83.9% under the naive reference. Mean gain +0.954 MOS; codec round-trip offset +0.14 MOS.
SSL-MOS20.5%SSL-MOS: hacked on 20.5% of utterances (mean of 5 attackers), 13.6% to 25.2% across attackers; 17.0% under the naive reference. Mean gain +0.191 MOS; codec round-trip offset −0.10 MOS.
DNSMOS OVRL13.6%DNSMOS OVRL: hacked on 13.6% of utterances (mean of 5 attackers), 6.4% to 20.0% across attackers; 18.3% under the naive reference. Mean gain +0.124 MOS; codec round-trip offset +0.02 MOS.
UTMOS225.7%UTMOS22: hacked on 5.7% of utterances (mean of 5 attackers), 4.2% to 7.6% across attackers; 9.4% under the naive reference. Mean gain +0.144 MOS; codec round-trip offset −0.07 MOS.
0255075100%
mean of five attackers, corrected referencerange across the fivemean, naive reference
Abstract
Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists.
The usual measurement has two flaws. First, the perturbation reaches the predictor through a processing chain
(here a neural codec) that shifts the score on its own, which scoring against the raw input charges to the
attack. Referencing the unperturbed round trip instead changes measured hackability by up to a factor of four
(0.31 to 0.08 for one defence). Second, one trained attacker is a sample, not a measurement: five attackers
differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence
claim needs the worst case over several. Under this protocol, four published predictors differ widely: NISQA
is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%. We then audit a closed
attack–detect–patch loop. It hardens the predictor only in its own attack space, by less than the
spread between attackers; a random-perturbation baseline matches it; and it costs up to 0.30 system SRCC out
of domain. Enhancers post-trained against patched predictors hack them far less (PESQ −0.03 versus
−0.23). Code, preregistration and run outputs are released.
Two corrections to the usual measurement
Attacks on quality predictors are usually scored by comparing the attacked audio with the
original recording, and reported for one trained attacker. Both choices change the answer.
Reference the codec round trip
NaiveR(y) − R(x) > τ
CorrectedR(y) − R(x̂) > τ
The perturbation reaches the predictor through a neural codec, and the codec moves the score on its own:
by −0.10 MOS for SSL-MOS, −0.07 for UTMOS, +0.02 for DNSMOS and +0.14 for NISQA, and by up to
+0.95 for a patched predictor. Scoring against the raw input x charges that offset to the attack.
Scoring against the zero-perturbation round trip x̂ moves the multi-space defence from
apparently catastrophic (0.306) to among the most robust (0.082).
Report the worst of several attackers
An attacker is itself a trained model. Five attackers that differ only in random seed reach success rates
from 0.000 to 0.261 against the undefended predictor R0 and from 0.000 to 0.382 against the
patched R3, a spread as large as every defence effect we measure.
Between predictors the spread is small relative to the differences (NISQA 0.876–0.916 against UTMOS
0.042–0.076), so the benchmark ranking holds where defence comparisons do not. We report the worst of
five attackers.
Attacker, detector and patch loop
A policy trained with group relative policy optimisation (GRPO) adds one offset to the codec latents of each
utterance, rescaled to a fixed size ε. It sees only the predictor's scores, so any predictor can be a
target. An output counts as a hack only if its score rises and at least one independent measure (ESTOI,
SI-SDR, SpeechBERTScore or the Whisper word error rate) says it got worse. No human labels are used. Each
hack becomes a preference pair x̂ ≻ y, and the predictor is fine-tuned on its
regression loss plus a margin ranking loss over every hack found so far.
Fig. 1. The attack–detect–patch loop. A GRPO policy perturbs codec latents
at a fixed size; an output is flagged only if its score rises above that of the zero-perturbation round trip
x̂ (dashed) and an independent panel degrades relative to x̂. Flagged pairs patch the
predictor; the next round retrains the attacker.
Listen to the attacks
Each attacked clip is paired with the reference its score is measured from: the codec round trip for
latent attacks, the clean input for waveform PGD. Switch between the two while a clip plays to hear what the
attack changed; most perturbations are small, so headphones help. A clip counts as hacked when the attacked
predictor scores it more than τ = 0.3 above the reference and at least one panel measure says it got
worse.
The clips are not from the paper’s test set, because BVCC may not be redistributed. The released
attackers, trained on BVCC, are applied unchanged to VoiceBank-DEMAND test speech, and every score is
computed on the exact file you hear.
Unchanged
Clean recording
the unchanged input
2.9 s
NISQA3.527
SSL-MOS4.219
DNSMOS OVRL3.364
UTMOS224.294
EnCodec round trip
encoded and decoded, no perturbation
2.9 s
NISQA4.865
SSL-MOS4.031
DNSMOS OVRL3.309
UTMOS224.024
DAC round trip
encoded and decoded, no perturbation
2.9 s
NISQA4.836
SSL-MOS4.062
DNSMOS OVRL3.394
UTMOS224.230
One kind of attack, four predictors
Each predictor is attacked by its own black-box policy in EnCodec latents (ε = 0.5), which sees only that predictor’s scores. Each card shows the attacked predictor’s score. On this clean read speech the rates are lower than Table I’s on BVCC, with the predictors in the same order.
NISQA
Not hacked
the strongest of its five attackers in Table I
2.9 s
NISQA score4.865→5.162+0.297
Score rises 0.297 over the EnCodec round trip, less than τ = 0.3. Measured from the clean input instead: +1.635.
Panel measures and files
vs. clean
round trip
attacked
ESTOI
0.962
0.935 (worse)
SI-SDR
9.0 dB
7.9 dB (worse)
WER
0%
0%
SBS
0.948
0.866 (worse)
Measured against the clean input. Red: worse than the reference by more than the panel’s margin.
Each predictor is attacked by its own black-box policy in EnCodec latents (ε = 0.5), which sees only that predictor’s scores. Each card shows the attacked predictor’s score. On this clean read speech the rates are lower than Table I’s on BVCC, with the predictors in the same order.
NISQA
Hacked
the strongest of its five attackers in Table I
2.7 s
NISQA score4.652→5.040+0.388
Score rises 0.388 over the EnCodec round trip, more than τ = 0.3, and the audio got worse by ESTOI, SI-SDR and SpeechBERTScore. Measured from the clean input instead: +0.328.
Panel measures and files
vs. clean
round trip
attacked
ESTOI
0.899
0.815 (worse)
SI-SDR
11.0 dB
9.3 dB (worse)
WER
0%
0%
SBS
0.934
0.853 (worse)
Measured against the clean input. Red: worse than the reference by more than the panel’s margin.
Each predictor is attacked by its own black-box policy in EnCodec latents (ε = 0.5), which sees only that predictor’s scores. Each card shows the attacked predictor’s score. On this clean read speech the rates are lower than Table I’s on BVCC, with the predictors in the same order.
NISQA
Not hacked
the strongest of its five attackers in Table I
4.2 s
NISQA score3.978→3.329−0.649
Score falls 0.649 below the EnCodec round trip. Measured from the clean input instead: −0.519.
Panel measures and files
vs. clean
round trip
attacked
ESTOI
0.657
0.527 (worse)
SI-SDR
14.9 dB
12.4 dB (worse)
WER
0%
0%
SBS
0.921
0.789 (worse)
Measured against the clean input. Red: worse than the reference by more than the panel’s margin.
Score rises 0.672 over the EnCodec round trip, more than τ = 0.3, and the audio got worse by ESTOI, SI-SDR and SpeechBERTScore. Measured from the clean input instead: +0.516.
Panel measures and files
vs. clean
round trip
attacked
ESTOI
0.657
0.476 (worse)
SI-SDR
14.9 dB
13.0 dB (worse)
WER
0%
0%
SBS
0.921
0.833 (worse)
Measured against the clean input. Red: worse than the reference by more than the panel’s margin.
Audio: VoiceBank-DEMAND clean test set, speakers p232 and p257, 16 kHz, by Cassia Valentini-Botinhao, University of Edinburgh (clean speech from VCTK) (source, CC BY 4.0); the attacked and decoded clips derived from it are shared under the same licence. Rates are over all 824 test utterances. The examples, 2.5 to 5 s long, are one per speaker drawn at random plus the utterance with the largest score gain among those the SSL-MOS attack hacks.
Common predictors differ by more than an order of magnitude
At one perturbation size NISQA is hacked on 89.6% of utterances for a mean gain of +0.954 MOS, SSL-MOS on
20.5%, DNSMOS OVRL on 13.6% and UTMOS on 5.7%. NISQA and DNSMOS are routinely used as reinforcement-learning
rewards, so the choice of reward is a first-order robustness decision, and the ranking is not predictable from
in-domain accuracy.
Predictor
offset
mean ASR
worst ASR
mean gain
SSL-MOS (BVCC)
−0.10
0.205
0.252
+0.191
UTMOS22
−0.07
0.057
0.076
+0.144
NISQA
+0.14
0.896
0.916
+0.954
DNSMOS OVRL
+0.02
0.136
0.200
+0.124
Table I. Hackability of common predictors at one size (ε = 0.5,
EnCodec latents), five independently trained attackers each, 500 BVCC-test utterances; “worst” is
the largest of the five. Offset is the codec round trip's own score change, R(x̂) −
R(x), in MOS.
What patching buys
We start from SSL-MOS trained on BVCC (R0) and run three rounds of the loop (R1 to
R3), a multi-space variant that also attacks DAC latents and the waveform, and three baselines that
patch on noise, waveform-PGD or random latent pairs with the same loss and step budget.
It hardens only its own attack space
EnCodec attack success falls from 0.226 (R0) to 0.050 (R1) and 0.017
(R2), but the R0→R3 difference does not hold its sign across seeds
(−0.073, −0.039, +0.379).
A random baseline matches it
Pairs from random latent perturbations of the same size, with no reinforcement learning and no detector,
give the lowest in-space success rate of any defence (0.019 ± 0.016).
Nothing transfers
Against an unseen DAC attacker the EnCodec-patched predictors are worse than undefended (0.065 rising to
between 0.13 and 0.16), and white-box PGD succeeds on nearly every utterance. Only multi-space training
resists DAC (0.016), and it is the one predictor PGD does not fully break (0.841).
Robustness costs agreement with listeners
Out of domain, mean system SRCC falls from 0.731 to 0.690, 0.682 and 0.637 over three rounds and to 0.552
for the multi-space predictor; in domain the ranking holds (0.920 vs 0.922) but utterance MSE rises from
0.234 to 0.272.
offset
ASR by attack space
human agreement
Predictor
R(x̂) − R(x)
EnCodec
DAC
PGD
BVCC
OOD
R0 (no defense)
−0.09±0.02
0.226±0.205
0.065±0.113
0.999±0.002
0.922±0.004
0.731±0.006
R1
−0.06±0.03
0.050±0.048
0.161±0.033
1.000±0.001
0.911±0.009
0.690±0.014
R2
0.00±0.07
0.017±0.006
0.141±0.025
1.000±0.000
0.918±0.003
0.682±0.008
R3 (EnCodec loop)
−0.27±0.25
0.137±0.179
0.127±0.031
1.000±0.000
0.920±0.001
0.637±0.046
R3multi (multi-space)
0.95±0.16
0.082±0.107
0.016±0.026
0.841±0.089
0.914±0.003
0.552±0.015
noise pairs
−0.09±0.01
0.270±0.107
0.230±0.028
1.000±0.000
0.917±0.004
0.722±0.013
PGD pairs
−0.08±0.02
0.223±0.057
0.225±0.044
0.969±0.025
0.909±0.001
0.635±0.009
random-latent pairs
−0.03±0.03
0.019±0.016
0.200±0.053
1.000±0.000
0.922±0.004
0.696±0.010
Table II. Auditing the loop. Corrected-reference attack success rate per
attack space (fresh attacker, BVCC test) and agreement with human MOS (BVCC system SRCC; out-of-domain mean
over 12 sets), mean ± sd over 3 seeds.
Fig. 2. (a) Benchmark of four common predictors at one perturbation size under the
naive (x) and corrected (x̂) ceiling references; error bars span the five attackers, and the
reference moves DNSMOS and NISQA in opposite directions. (b) In-loop attack success rate (BVCC train,
corrected reference) of the round-k attacker against the predictor patched in round k − 1,
one line per seed.
Downstream, patched predictors are safer rewards
A small speech enhancer, pre-trained on VoiceBank-DEMAND, is post-trained with GRPO using one predictor's
score as its only reward. Against R0 it shows the textbook hack: the reward rises by +0.263 while
PESQ falls by 0.226, ESTOI by 0.057 and SI-SDR by 6.1 dB. Against R3 the reward rises as much
(+0.269) with PESQ intact (−0.034), and the random-pair baseline again matches it.
Measure
R0
R3
random-latent pairs
Δ own reward
0.263±0.010
0.269±0.028
0.334±0.051
ΔPESQ
−0.226±0.170
−0.034±0.126
0.019±0.050
ΔESTOI
−0.057±0.010
−0.038±0.013
−0.043±0.006
ΔSI-SDR (dB)
−6.1±1.4
−5.8±1.4
−5.7±0.6
ΔDNSMOS
−0.075±0.031
−0.076±0.056
−0.075±0.043
ΔNISQA
0.335±0.186
0.300±0.072
0.468±0.059
ΔUTMOS
0.018±0.006
0.057±0.030
0.076±0.011
Table III. Downstream GRPO enhancement. Change against the supervised
pre-trained enhancer on VoiceBank-DEMAND test, per reward predictor. Mean ± sd over 3 seeds.
Limitations
The attacker is black-box and mild. Its hacks against SSL-MOS move ESTOI by −0.023 and SI-SDR by
−1.034 dB, and two independent judges call them perceptually marginal (UTMOS −0.004, DNSMOS
−0.048; PGD moves UTMOS by +0.802). The benchmark ranks predictors against this attack family only.
We ran no listening test.
The detector is not fully independent: two panel measures are neural, and for waveform attacks its
reference is the input itself, so success rates are not comparable across attack families.
The patch study uses 3 seeds and one predictor family (46 A100-hours in total).
Reproduce every number
The release holds every checkpoint behind the paper (195 files, 19.3 GB: predictors, every patch round, all
attackers and enhancers), the exact configuration of each run, the per-run outputs and the preregistration.
Each table, figure and result has its own notebook. A notebook either re-runs the evaluation from the released
checkpoints on a GPU, or aggregates the released run outputs on a CPU in seconds, and then compares its numbers
with the paper's.
# environment (Python 3.11)
git clone https://github.com/saliAlavi/audio_assessment_hackability
cd audio_assessment_hackability
pip install -r requirements.txt
pip install --no-deps -r requirements-nodeps.txt
export SCRATCH_DIR=/path/with/space # checkpoints, caches, runs
source scripts/env.sh
# every number in the paper from the released outputs (CPU)
NB_SOURCE=released jupyter nbconvert --to notebook --execute \
notebooks/results/paper_numbers.ipynb
# or re-run the evaluations from checkpoints (77 jobs, ~17 A100-hours)
NB_SOURCE=recompute NB_RUN=local jupyter nbconvert --to notebook \
--execute notebooks/results/paper_numbers.ipynb
@misc{alavi2026hackable,
title = {How Hackable Is Your Speech Quality Metric? A Corrected Protocol,
a Benchmark, and What Patching Buys},
author = {Alavi, Ali and Williamson, Donald S.},
year = {2026},
eprint = {2610.10899},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2610.10899}
}
