The raw data for three metrics
The 13 metrics are WER, CER, normalized variants of WER, embedding-based metrics, and several published metrics that use an LLM. Each one scored the same 2,006 transcripts that the native speakers marked by hand. The scatter plots show every transcript for the three main metrics. A good metric puts the dots in a narrow band that rises from left to right: when the annotators find more errors, the metric does too.
LLM judge (SER) · ρ = 0.71
WER · ρ = 0.52
CER · ρ = 0.51
2,006 transcripts. The horizontal position is the annotator's SER; the vertical position is the metric. On the dashed line, the metric equals the annotator. ρ is the Spearman correlation with the annotators. The dots at the far left are the transcripts with no errors: the judge keeps most of them near zero, while WER spreads 80% of them between 15.8% and 53.8%.
The LLM judge’s dots form a band along the dashed line. The WER dots form a wide cloud far above it. For transcripts with no errors, WER ranges from about 13% to 60% (5th to 95th percentile), and for transcripts with 10–20% real errors, from about 29% to 66%. The two ranges overlap almost completely.
The ranking
The correlation measures how narrow the band is. This is the ranking of all 13 metrics by that correlation:
ρ is the Spearman correlation between each metric and the SER of the annotators on 2,006 pairs. A higher value is better. MAE is the average difference in percentage points. A lower value is better. A metric with the name proxy is our Darija version of a published metric. It is not the original software.
What the ranking shows
- Asking an LLM directly about meaning is what matters. The LLM judge reaches 0.71; we use SER because the judge marks the exact words it considers wrong, so every decision can be checked.
- Using an LLM only to correct WER is not enough. LLM-WER, SN-WER, and OIWER-WER keep WER’s word alignment at their core, and they reach between 0.35 and 0.60.
- A small average distance does not mean a good ranking. CER is much closer to the annotators than WER on average (7.4 points), but its correlation, 0.51, is almost the same as WER’s. Embedding metrics have lower correlations.
A correlation is one number for the whole study. The parent page breaks the comparison into three tests that match how a metric is used.