Six kinds of spelling difference

These patterns came up again and again in our study with native speakers. None of them changes the meaning, but each one adds at least one error to WER.

1. Attached or separate words
Short words can attach to the next word or stand apart: فداركم and في داركم. One word becomes two, and WER counts a substitution plus an insertion.
2. Script of borrowed words
ستانفورد and Stanford. Both are correct; which one you get depends on the writer’s convention.
3. Numbers
أربعين and 40. A number written out can take several words, while the digits are a single word, so one number can cost several errors.
4. Diacritics
Short vowel marks are optional in Arabic script. A system that adds them to every word can score close to 100% WER on a sentence that is entirely correct.
5. Different letters for one sound
The /g/ sound can be written ݣ, ك, or ق. Hamza forms vary (أكلة and اكلة), and words can end in ة or ه.
6. Spelling by ear
Vowel spelling and dialect forms change from speaker to speaker: ما عندوش and معندوش.

The figure shows six transcripts from the study. Switch between views to see which words each method marks as errors.

One transcript, three methods of evaluation

The hypothesis writes some attached words as separate words (فداركم is written في داركم). It also removes some hamzas. The meaning does not change.

Human SER

0%

LLM SER

3.8%

WER

82.3%

CER

17.3%

Words that WER counts as errors

Reference

وحنا محتاجين مرا متعلمة، ولكن التعلم ماشي معناه هو الاختلاط. اليوم راه يمكن ليك تقرا فداركم فستانفورد، وكامبريدج، وإم آي تي، وأوبن كورس وير، وإيديكس، وكورسيرا، وݣاع المنصات العالمية بأرقى الشواهد وأنت جالس فدارك.

Hypothesis · Gemma-4-E4B

و حنا محتاجين مرا متعلمة ولكن التعلم ماشي المعنى هو الاختلاط اليوم راه يمكن ليك تقرا في داركم في ستانفورد و كامبريدج و ام آي تي و اوبن كورس وير و ايديكس و كورسيرا وكاع من المنصات العالمية بارقى الشواهد و نتا جالس في دارك

These pairs come from our study with native speakers. The reference is the human transcript. The hypothesis is the transcript of an ASR system. The human and LLM scores are SER. Select a view to see the words that each method marks as errors.

WER is still useful. It is fast and deterministic, and it exposes large failures: on one clip, MoulSot wrote the syllable “gu” about 8,000 times, and WER shows this immediately. That is why we report WER and CER next to the meaning-error rate.

How big the effect is

In our study, native speakers found no errors at all in 497 of the 2,006 transcripts (24.8%). On exactly those transcripts, the average WER is 34%.

Averaged per system, WER sits 32.9 points above the annotators’ score. CER sits 4.6 points above it, and the LLM judge 1.9 points.

The LLM judge labels every difference it finds, which shows where this excess comes from. Most differences between a hypothesis and its reference are variations of the kinds listed above:

What the judge accepts, and how often

All differences between hypotheses and references

9,802 variations (59%)
6,693 errors (41%)

WER counts all of them. SER counts only the errors.

The variations, by type

Dialect forms
4,019
لي ↔ الليف ↔ فيولي ↔ والليبلي ↔ باللي
Orthography: hamza, spacing, attached words
3,441
إلا ↔ إلىإيلا ↔ إلاكانو ↔ كانوارّيتم ↔ الريتم
Script and number changes
1,369
2 ↔ جوج1 ↔ واحددونك ↔ doncجيم ↔ j'aime
Spelling
973
هدا ↔ هذاهادا ↔ هذاهدا ↔ هاداوهدا ↔ وهذا

Counts over the 2,006 transcripts of the study, graded by the main judge (GPT-5.6 Luna). Each difference is one place where the hypothesis and the reference are written differently; it can cover more than one word. The type labels are the judge's own, and they overlap: the same change can be called "dialect" in one transcript and "spelling" in another.

The offset isn’t the same for every system. It depends on how closely a system’s spelling habits happen to match those of the person who wrote the reference. So WER doesn’t just add a constant; it rewards systems for matching a spelling style, which has nothing to do with hearing correctly.

Can normalization fix it?

Normalization rewrites both texts into a common form before computing WER. We tried three approaches: rules that strip diacritics and punctuation and unify letter forms; a list of accepted spelling variants, in the spirit of WERd for Egyptian Arabic; and an LLM that rewrites both texts into one spelling before WER is computed (SN-WER).

How far normalization gets you

WER

the words exactly as written

ρ 0.52 · dist. 33.2

Norm-WER

after rules: no diacritics or punctuation, unified letter forms

ρ 0.53 · dist. 22.8

WERd-proxy

with a list of accepted spelling variants, in the spirit of WERd

ρ 0.35 · dist. 9.4

SN-WER

after an LLM rewrites both texts into one spelling

ρ 0.60 · dist. 23.7

LLM SER

an LLM judges the meaning directly

ρ 0.71 · dist. 4.8

Spearman correlation with the annotators' SER on the 2,006 transcripts of the study (1 means perfect agreement). The average distance is the mean absolute difference from the annotators' SER, in points. Each normalization step changes the text before WER counts edits; only the last row stops counting edits.

The rules help only a little. Correlation with the annotators goes from 0.52 to 0.53, and the average distance from their score stays above 20 points. Rules can’t handle attached words, script changes, numbers, or spelling by ear without also merging words that really are different.

The variant list brings WER much closer to the annotators on average (9.4 points), but its correlation drops to 0.35: it accepts some differences that do change the meaning.

The LLM rewrite does better, raising the correlation to 0.60, but WER computed on top still over-counts. What works is to stop counting edits and ask directly whether the meaning changed. That is what the LLM judge does, and the next page shows how we checked it.