The study with native speakers

We took 507 clips from the test set and the transcripts that 5 ASR systems produced for them: Gemini, an earlier version of our 1.7B model, MoulSot, and two sizes of Gemma 4. That gave 2,006 pairs, each made of one reference and one hypothesis.

7 native Darija speakers shared the work, without knowing which system wrote which hypothesis. Rather than giving a score, they marked the words in error: in the reference, words the hypothesis lost or changed; in the hypothesis, words that were added or wrong. Differences in spelling, script, spacing, and dialect did not count as errors.

Marking words is more concrete than giving an overall score, and the LLM judge produces exactly the same kind of output. That lets us score both with one formula:

SER=100×nreferr+nhyperrWref+Whyp\mathrm{SER} = 100 \times \frac{n^{\text{err}}_{\text{ref}} + n^{\text{err}}_{\text{hyp}}}{W_{\text{ref}} + W_{\text{hyp}}}

Counting errors on both sides matters. A dropped phrase only appears in the reference, and an invented one only in the hypothesis, so a metric based on the reference alone can’t see invented text.

What the judge does with one transcript

The figure below follows one real pair through the judge, step by step, and compares the result with WER. The hypothesis has five differences from the reference. Four of them are ways of writing the same thing; one changes a word.

How the judge scores one transcript
1

The judge receives the reference and the hypothesis

Reference

إلا فاتت الصداقة 7 سنين، را عندها نسبة 90 فالمية أنها غتبقى للأبد.

Hypothesis

إلى فاتت الصدقة سبع سنين راه عندها نسبة 90% أنها غتبقى للأبد

Meaning: “If a friendship lasts 7 years, it has a 90 percent chance of lasting forever.”

2

WER counts every difference

Reference

إلا فاتت الصداقة 7 سنين، را عندها نسبة 90 فالمية أنها غتبقى للأبد.

Hypothesis

إلى فاتت الصدقة سبع سنين راه عندها نسبة 90% أنها غتبقى للأبد

8 word edits for 13 reference words: WER = 61.5%. A missing comma or full stop also counts, because WER compares the words exactly as written.

3

The judge sorts each difference into variation or error

Reference

إلا فاتت الصداقة 7 سنين، را عندها نسبة 90 فالمية أنها غتبقى للأبد.

Hypothesis

إلى فاتت الصدقة سبع سنين راه عندها نسبة 90% أنها غتبقى للأبد

error: the meaning changes variation: same meaning
  • errorالصداقة → الصدقة · The hypothesis changes 'friendship' to 'charity/alms', altering the meaning.
  • orthographyإلا → إلى
  • code-switch7 → سبع
  • dialectرا → راه
  • code-switch90 فالمية → 90%
4

SER counts only the words inside errors, on both sides

SER=100×1+113+12=8.0%\mathrm{SER} = 100 \times \frac{1 + 1}{13 + 12} = 8.0\%

Words in errors: 1 in the reference and 1 in the hypothesis, out of 13 and 12 words.The annotator’s SER for this pair is also 8%. WER is 61.5%.

A real pair from the study. The hypothesis was produced by Gemma-4-E4B. The annotator also found one error in it, for a human SER of 8%.

WER and SER differ in two ways. WER counts every difference, while SER counts only the differences that change the meaning. And WER divides by the reference length, while SER divides by the length of both texts, so a missing word and an invented word weigh the same.

What the judge accepts

Across the whole study, most differences between hypothesis and reference are variations. The figure counts them by type, with the most common examples of each.

What the judge accepts, and how often

All differences between hypotheses and references

9,802 variations (59%)
6,693 errors (41%)

WER counts all of them. SER counts only the errors.

The variations, by type

Dialect forms
4,019
لي ↔ الليف ↔ فيولي ↔ والليبلي ↔ باللي
Orthography: hamza, spacing, attached words
3,441
إلا ↔ إلىإيلا ↔ إلاكانو ↔ كانوارّيتم ↔ الريتم
Script and number changes
1,369
2 ↔ جوج1 ↔ واحددونك ↔ doncجيم ↔ j'aime
Spelling
973
هدا ↔ هذاهادا ↔ هذاهدا ↔ هاداوهدا ↔ وهذا

Counts over the 2,006 transcripts of the study, graded by the main judge (GPT-5.6 Luna). Each difference is one place where the hypothesis and the reference are written differently; it can cover more than one word. The type labels are the judge's own, and they overlap: the same change can be called "dialect" in one transcript and "spelling" in another.

Six differences in ten are variations. WER counts every one of them as an error, which is why it is so much higher than the annotators’ SER.

How closely each metric follows the annotators

We compared 13 automatic metrics with the annotators’ SER on every pair. The LLM judge reaches a correlation of 0.71, the highest of the 13. WER reaches 0.52 and CER 0.51. Metrics that use an LLM only to correct WER stay between 0.35 and 0.60. The scatter plots, the ranking of all 13 metrics, and what we conclude from them are on their own page.

Three tests, not one average

WER is about 33 points higher than the annotators’ SER on average, but a high average is not the problem: if every score were 33 points too high, we could subtract 33. A correlation is not the whole answer either, because it summarizes agreement in one number. The useful question is whether a metric follows the annotators in the ways a metric is used. We ran three tests:

Three tests of a metric, scored against native speakers

1.Score a single transcript

LLM judge
0.58
WER
0.29
CER
0.23

R² is the share of the variation in the annotators' SER that the metric accounts for (the squared Pearson correlation). 1 means the metric moves exactly with the annotators; 0 means it is unrelated.

2.Recognize a correct transcript

LLM judge
68%
29%
WER
18%
82%
score 0–5% score 5–20% score above 20%

The 497 transcripts in which the annotators found no errors, split by the score each metric gives them. WER gives a non-zero score to 99.8% of them.

3.Measure the gap between two systems

LLM judge
1.1 pts
WER
3.7 pts
CER
3.9 pts
Overcount per systemAnnotatorsLLM judgeWERCER
Gemini3.3%+0.8+28.2+5.2
fihrIA-ASR (earlier 1.7B)5.4%+2.0+32.4+8.4
MoulSot8.8%+3.2+36.2+7.0
Gemma-4-E4B12.8%+1.7+34.5+2.1
Gemma-4-E2B14.6%+2.5+34.9+1.0
Range of the overcount2.48.07.4

For each of the 10 pairs of systems, the gap the metric reports minus the gap the annotators found, averaged in absolute value. Lower is better. The gaps go wrong because WER's overcount depends on the system's writing style, as the table shows.

All tests use the 2,006 transcripts of the study, each marked by a native speaker. A second LLM judge (Gemini 3.8 Flash) gives the same results: R² 0.59 in test 1.

1. Scoring a single transcript. The judge’s score accounts for 58% of the variation in the annotators’ SER, against 29% for WER and 23% for CER. Normalizing the text before computing WER does not close the gap (see the page on spelling variation).

2. Recognizing a correct transcript. On the 497 transcripts with no errors, WER is above zero 99.8% of the time and above 20% 82% of the time, depending on how many spelling differences the clip has. The judge gives 68% of them a score of 5% or less. A WER of 30% can therefore describe a perfect transcript or a poor one.

3. Measuring the gap between two systems. This is what a model comparison reports. Over the 10 pairs of systems, the gap the judge reports is on average 1.1 points from the gap the annotators found; for WER it is 3.7 points, and for CER 3.9. The reason is that WER’s overcount depends on the writing style of the system: it is 28.2 points for Gemini and 36.2 points for MoulSot. That range is larger than the gap between any two neighbouring systems in the study (1.8 to 3.9 points). For example, the annotators find 3.8 points fewer errors in MoulSot than in Gemma-4-E4B. WER shows a gap of only 2.2 points and needs about 132 test clips to detect it reliably; the LLM judge needs about 49. CER ranks the two systems in the wrong order. The overcount of the LLM judge stays between 0.8 and 3.2 points for all five systems.

The judge is ahead in all three tests. It is far ahead when a score is read on its own, for one transcript or one system, because WER’s score then includes all the spelling differences of that clip or that system.

Test 2 is easiest to understand from the score distributions. A metric that separates correct and flawed transcripts gives the two groups different scores, so their histograms overlap little:

Scores of correct and flawed transcripts

LLM judge (SER) · AUC = 0.82

0%20%40%60%80%100%

WER · AUC = 0.71

0%20%40%60%80%100%
no errors (annotator) at least one error

Green: the 497 transcripts with no errors according to the annotators. Blue: the 1,509 transcripts with at least one error. Each bar is the share of its group in a 5-point range of the metric. Where the two colors overlap, the metric cannot tell the groups apart. The AUC is the chance that a random flawed transcript scores worse than a random correct one.

The judge gives 87% of the correct transcripts a score below 10%. The flawed transcripts that also score below 10% are mostly transcripts with small errors: the annotators’ median SER for them is 5.6%. Under WER, the correct transcripts sit in the middle of the flawed ones, so a WER of 35% says almost nothing about whether a transcript has real errors.

Results for each system

The same order, but different distances
Human SER LLM SER WER
Gemini
fihrIA-ASR v0.5 (1.7B)
MoulSot
Gemma-4-E4B
Gemma-4-E2B
0%10%20%30%40%50%

Five systems from the study with native speakers. LLM SER and WER put the systems in the same order as the annotators. On average, WER is 32.9 points above the annotators, and LLM SER is 1.9 points above. For the difference between two systems, the average error is 1.1 points for LLM SER and 3.7 points for WER.

All three metrics put the five systems in the same order as the annotators. They differ in the distances. For each of the ten pairs of systems, we compared the gap a metric reports with the gap the annotators found. On average, the LLM judge is off by 1.1 points and WER by 3.7 points.

The scale of the scores

The judge’s scores are also on the same scale as the annotators’. The figure groups the pairs by the annotators’ SER and shows the average of each metric in each group.

Does the metric increase by the same quantity as the human SER?
0%20%40%60%80%0%n=4970–5%n=3785–10%n=40310–20%n=46520–40%n=24240+%n=21Human semantic error rate (group)WERCERLLM SERPerfect agreement

We put the 2,006 pairs into groups by the SER of the annotators. Each point is the average value of one group. The dashed line shows a metric that agrees perfectly with the annotators.

The LLM judge follows the annotators at every level. WER is about 33 points higher at every level. On the 24.8% of transcripts with no errors, WER averages 34% and the LLM judge 4.2%. A constant offset alone would not make WER a bad metric, but it means that a WER value cannot be read as a share of wrong words. An SER can: because the judge is on average only 1.9 points above the annotators, an SER of 5.8% means that about 6 words in 100 carry a meaning error.

Where the judge gets it wrong

The judge is a little too strict at times. On transcripts the annotators found error-free, it gives an average SER of 4.2%, and more than 5% in 32% of them. Technical vocabulary that mixes scripts is a typical trigger. On one agriculture clip, the hypothesis writes l’azote and le potassium in Latin script. The annotator found nothing wrong; the judge accepted the script and number changes but still flagged a few word choices, for an SER of 8.5%.

It is occasionally too lenient as well. In the “Diacritics” example of the article’s six transcripts, the annotator found one real error that the judge scored as 0%.

The human labels aren’t perfect either. Each pair was annotated by one person, so we can’t separate the judge’s mistakes from ordinary disagreement between annotators. In some cases the judge may be right and the annotator too lenient.

Avoiding a circular evaluation

Gemini wrote our training labels, and a Gemini judge might prefer transcripts in Gemini’s style. So our main judge is GPT-5.6 Luna, from a different company.

To test the concern, we re-ran the whole study with a Gemini judge and the same prompt. It reaches a correlation of 0.72 with the annotators and 0.88 with the GPT judge. It is slightly stricter: it is 3.9 points above the annotators, against 1.9 for GPT. Both judges recover the annotators’ ranking. We still keep the GPT judge as the main one, because it is the cleaner design. On the full test set we went further and graded every transcript with four judges; see the effect of the judge model.

Further in