The test

Every system transcribed the same 508 test clips, about 2 hours of audio. None of the clips come from a video in the training data. Four LLM judges graded every transcript, and we also computed WER and CER. Use the buttons to change the metric or the judge.

Results of 9 systems on the same 508 test clips (a lower value is better)
  • Gemini 3.8 Flash
    5.0%
  • fihrIA-ASR 1.7B
    6.4%
  • fihrIA-ASR 0.6B
    7.6%
  • Voxtral Mini 4B
    8.4%
  • MoulSot
    11.2%
  • Gemma-4-E4B
    15.3%
  • Gemma-4-E2B
    18.9%
  • GPT-Audio-Mini
    23.2%
  • Inkling-Small
    36.6%

SER is the percentage of words that are in meaning errors. This view is the mean of GPT-5.6 Luna, Gemini 3.8 Flash, Claude Sonnet 5, and GLM-5.3. On that mean, both of our models are ahead of every other open-weight system, followed by Voxtral Mini 4B and MoulSot. The other buttons show each judge on its own.

On the mean of the four judges, our 1.7B model makes meaning errors on 6.4% of words and our 0.6B model on 7.6%. Gemini 3.8 Flash, a closed commercial system, is ahead of both, at 5.0%. Among open-weight systems, both our 1.7B and 0.6B models are ahead of Voxtral Mini 4B (8.4%) and MoulSot (11.2%).

Cross-judge agreement

All four judges unanimously rank our 1.7B model first among open-weight systems. GPT-5.6 Luna, Gemini 3.8 Flash, and Claude Sonnet 5 also rank our 0.6B model ahead of Voxtral (7.5%, 9.8%, and 9.7%), while GLM-5.3 puts Voxtral (6.5%) between our 1.7B (5.8%) and 0.6B (7.2%) models.

Across every judge, both of our models stay well ahead of MoulSot. The page on the judge model has every judge’s score for every system.

Word and character error

We report the median over clips, because one very bad clip can dominate a total. MoulSot wrote the syllable “gu” about 8,000 times on one clip, which takes its total WER to 91.1% while its median is 42.3%.

  • 1.7B. Median WER 30.9%, against 36.5% for Voxtral and 42.3% for MoulSot. Median CER 10.3%, against 10.6% and 13.0%.
  • 0.6B. Median WER 31.6% and median CER 10.7%.

Both of our models have a lower median WER than Voxtral and MoulSot. On median CER, our two models and Voxtral are within half a point of each other. These values are high for every system for the reason given in the page on spelling variation: the references and the systems do not spell Darija the same way.

Choosing between the two sizes

  • 1.7B, the most accurate. At the same size as MoulSot, it makes 1.7 times fewer meaning errors (6.4% of words against 11.2%).
  • 0.6B, the most efficient. With 2.8 times fewer parameters than MoulSot and 6.7 times fewer than Voxtral, it makes 1.5 times fewer meaning errors than MoulSot (7.6% against 11.2%) and 0.9 times as many as Voxtral (8.4%).

The two models are 1.2 points apart in meaning errors. Use the 1.7B model when accuracy matters most. Use the 0.6B model when speed or cost matters, or to run on a laptop.

Speed

On one H100 with a batch of 32 clips, our 0.6B model transcribes the test set at 255× real time and our 1.7B model at 243×. Voxtral Mini 4B runs at 34×, so our models are 7.5 and 7.1 times faster.

Against MoulSot, at the same precision:

  • Our 1.7B model is 1.65 times faster. It has the same size as MoulSot, so the gain comes from the Darija tokenizer alone: it writes a Darija word in 1.32 tokens on average instead of 2.29, so the decoder runs fewer steps.
  • Our 0.6B model is 2 times faster. It adds a smaller size to the Darija tokenizer: a third of MoulSot’s parameters. This figure is the laptop comparison, both models in 4-bit.

The tokenizer does not cost accuracy. A control model identical to ours except for the original Qwen tokenizer has the same WER within 0.3 points and the same SER within 0.1 points. The tokenizer page has that comparison, and the laptop page shows where the laptop speed comes from.

Further in