Six kinds of spelling difference
These patterns came up again and again in our study with native speakers. None of them changes the meaning, but each one adds at least one error to WER.
- 1. Attached or separate words
- Short words can attach to the next word or stand apart: فداركم and في داركم. One word becomes two, and WER counts a substitution plus an insertion.
- 2. Script of borrowed words
- ستانفورد and Stanford. Both are correct; which one you get depends on the writer’s convention.
- 3. Numbers
- أربعين and 40. A number written out can take several words, while the digits are a single word, so one number can cost several errors.
- 4. Diacritics
- Short vowel marks are optional in Arabic script. A system that adds them to every word can score close to 100% WER on a sentence that is entirely correct.
- 5. Different letters for one sound
- The /g/ sound can be written ݣ, ك, or ق. Hamza forms vary (أكلة and اكلة), and words can end in ة or ه.
- 6. Spelling by ear
- Vowel spelling and dialect forms change from speaker to speaker: ما عندوش and معندوش.
The figure shows six transcripts from the study. Switch between views to see which words each method marks as errors.
The hypothesis writes some attached words as separate words (فداركم is written في داركم). It also removes some hamzas. The meaning does not change.
Human SER
0%
LLM SER
3.8%
WER
82.3%
CER
17.3%
Words that WER counts as errors
Reference
وحنا محتاجين مرا متعلمة، ولكن التعلم ماشي معناه هو الاختلاط. اليوم راه يمكن ليك تقرا فداركم فستانفورد، وكامبريدج، وإم آي تي، وأوبن كورس وير، وإيديكس، وكورسيرا، وݣاع المنصات العالمية بأرقى الشواهد وأنت جالس فدارك.
Hypothesis · Gemma-4-E4B
و حنا محتاجين مرا متعلمة ولكن التعلم ماشي المعنى هو الاختلاط اليوم راه يمكن ليك تقرا في داركم في ستانفورد و كامبريدج و ام آي تي و اوبن كورس وير و ايديكس و كورسيرا وكاع من المنصات العالمية بارقى الشواهد و نتا جالس في دارك
These pairs come from our study with native speakers. The reference is the human transcript. The hypothesis is the transcript of an ASR system. The human and LLM scores are SER. Select a view to see the words that each method marks as errors.
WER is still useful. It is fast and deterministic, and it exposes large failures: on one clip, MoulSot wrote the syllable “gu” about 8,000 times, and WER shows this immediately. That is why we report WER and CER next to the meaning-error rate.
How big the effect is
In our study, native speakers found no errors at all in 497 of the 2,006 transcripts (24.8%). On exactly those transcripts, the average WER is 34%.
Averaged per system, WER sits 32.9 points above the annotators’ score. CER sits 4.6 points above it, and the LLM judge 1.9 points.
The LLM judge labels every difference it finds, which shows where this excess comes from. Most differences between a hypothesis and its reference are variations of the kinds listed above:
All differences between hypotheses and references
WER counts all of them. SER counts only the errors.
The variations, by type
Counts over the 2,006 transcripts of the study, graded by the main judge (GPT-5.6 Luna). Each difference is one place where the hypothesis and the reference are written differently; it can cover more than one word. The type labels are the judge's own, and they overlap: the same change can be called "dialect" in one transcript and "spelling" in another.
The offset isn’t the same for every system. It depends on how closely a system’s spelling habits happen to match those of the person who wrote the reference. So WER doesn’t just add a constant; it rewards systems for matching a spelling style, which has nothing to do with hearing correctly.
Can normalization fix it?
Normalization rewrites both texts into a common form before computing WER. We tried three approaches: rules that strip diacritics and punctuation and unify letter forms; a list of accepted spelling variants, in the spirit of WERd for Egyptian Arabic; and an LLM that rewrites both texts into one spelling before WER is computed (SN-WER).
WER
the words exactly as written
Norm-WER
after rules: no diacritics or punctuation, unified letter forms
WERd-proxy
with a list of accepted spelling variants, in the spirit of WERd
SN-WER
after an LLM rewrites both texts into one spelling
LLM SER
an LLM judges the meaning directly
Spearman correlation with the annotators' SER on the 2,006 transcripts of the study (1 means perfect agreement). The average distance is the mean absolute difference from the annotators' SER, in points. Each normalization step changes the text before WER counts edits; only the last row stops counting edits.
The rules help only a little. Correlation with the annotators goes from 0.52 to 0.53, and the average distance from their score stays above 20 points. Rules can’t handle attached words, script changes, numbers, or spelling by ear without also merging words that really are different.
The variant list brings WER much closer to the annotators on average (9.4 points), but its correlation drops to 0.35: it accepts some differences that do change the meaning.
The LLM rewrite does better, raising the correlation to 0.60, but WER computed on top still over-counts. What works is to stop counting edits and ask directly whether the meaning changed. That is what the LLM judge does, and the next page shows how we checked it.