Every judge, every system

Each judge graded all 9 systems on the 508 test clips. The figure draws one line per system across the four judges.

Four judges, 9 systems
4%6%10%15%25%40%GLM-5.3GPT-5.6 LunaGemini 3.8 FlashClaude Sonnet 5Gemini 3.8 Flash · 5.7%fihrIA-ASR 1.7B · 7.0%fihrIA-ASR 0.6B · 8.0%Voxtral Mini 4B · 9.7%MoulSot · 13.4%Gemma-4-E4B · 16.4%Gemma-4-E2B · 20.0%GPT-Audio-Mini · 24.2%Inkling-Small · 39.4%

Each line is one system, and each column is one judge. The vertical axis is logarithmic, so equal distances mean equal ratios. All four judges agree on the order of eight of the nine systems, placing our 1.7B model first among open-weight systems; only GLM ranks Voxtral Mini 4B between our 1.7B and 0.6B models.

Two things are visible at once. Every judge preserves the overall ordering across eight of the nine systems, with our 1.7B model first among all open-weight systems under all four judges. Only Voxtral Mini 4B shifts slightly: GPT-5.6 Luna, Gemini 3.8 Flash, and Claude Sonnet 5 all place it behind both of our models, whereas GLM-5.3 places it between our 1.7B and 0.6B models. The table gives the exact values; the number in parentheses is the system’s position under that judge.

SystemGPT-5.6 LunaGemini 3.8 FlashClaude Sonnet 5GLM-5.3
Gemini 3.8 Flash4.3 (1)5.2 (1)5.7 (1)4.6 (1)
fihrIA-ASR 1.7B5.8 (2)7.1 (2)7.0 (2)5.8 (2)
fihrIA-ASR 0.6B6.9 (3)8.2 (3)8.0 (3)7.2 (4)
Voxtral Mini 4B7.5 (4)9.8 (4)9.7 (4)6.5 (3)
MoulSot10.2 (5)11.9 (5)13.4 (5)9.2 (5)
Gemma-4-E4B14.7 (6)16.4 (6)16.4 (6)13.8 (6)
Gemma-4-E2B18.5 (7)20.1 (7)20.0 (7)17.1 (7)
GPT-Audio-Mini22.4 (8)25.2 (8)24.2 (8)21.1 (8)
Inkling-Small36.1 (9)37.6 (9)39.4 (9)33.2 (9)

What changes and what doesn’t

The top open-weight ranking never changes. All four judges put Gemini 3.8 Flash first, then our 1.7B model first among all open-weight systems, and MoulSot, the Gemma models, and the remaining closed systems behind. Only the boundary between our 0.6B model and Voxtral Mini 4B shifts: GPT, Gemini, and Claude place our 0.6B model ahead of Voxtral, whereas GLM places Voxtral between our two models.

The absolute values vary moderately across judges. Gemini 3.8 Flash and Claude Sonnet 5 are slightly stricter than GPT-5.6 Luna and GLM-5.3, with spreads of roughly 1.3 to 3.3 points on the top systems. An SER figure is therefore most informative when accompanied by the judge name or a multi-judge mean.

The gap to MoulSot holds under every judge. MoulSot, the other open-weight Qwen3-ASR fine-tune for Darija, makes 1.6 to 1.9 times as many meaning errors as our 1.7B model, and 1.3 to 1.7 times as many as our 0.6B model, depending on the judge.

Agreement on single transcripts

The same ranking could come from judges that disagree on many single transcripts but average out. To check this, we compared two judges transcript by transcript, on the study with native speakers.

Two judges grading the same transcripts
00202040406060GPT-5.6 Luna SER (%)Gemini 3.8 Flash SER (%)

Each dot is one transcript of the study with native speakers (2,006 in total), graded by both GPT-5.6 Luna and Gemini 3.8 Flash. Most dots lie close to the dashed line of perfect agreement, with a Spearman correlation of 0.88. They lie a little above it on average, because Gemini is slightly stricter.

The two judges agree closely on single transcripts too. The median difference between their scores is 2.5 points, and only 5% of the transcripts differ by more than 10 points. In only 15 of the 2,006 transcripts does one judge find no errors while the other gives an SER above 10%.