The two test sets

DarijaCS has two parts, and they are scored separately.

  • Prism (a.k.a. ISMA), 357 clips and about 90 minutes of audio. Each clip has a subject (tourism, finance, physics, engineering, and others), and most have a noise level and a speaking pace.
  • Probe (a.k.a. Darija in the Wild), 109 clips and about 15 minutes of audio. Most clips carry a label for what makes them hard: code-switching, technical vocabulary, noise, speed, or silence.

How we score

We follow the protocol that Voxtral describes for its own figures:

  • Greedy decoding at temperature 0.
  • Voxtral’s Arabic text normalizer, applied to the reference and to the hypothesis.
  • Character error rate and word error rate over the whole set: substitutions, deletions, and insertions, divided by the length of the references.
  • Prism and Probe scored separately, never merged.

The error rates here are much lower than the medians in the article. The clips, the references, and the normalizer are all different, so the two sets of numbers are not comparable. Only the order of the systems is.

Voxtral’s published figures next to ours

Voxtral’s chart reports character error for Voxtral Mini 4B Realtime Arabic: 6.68% on Prism (a.k.a. ISMA) and 9.69% on Probe (a.k.a. Darija in the Wild). The other bars are our measurements on the released clips, including a run of Voxtral we could not bring down to those published figures.

Character error (lower is better)
  • fihrIA-ASR
  • Voxtral, as published
  • Voxtral, our run
  • MoulSot

Prism (a.k.a. ISMA)

  • Our 1.7B
    6.05%
  • Voxtral, published
    6.68%
  • Our 0.6B
    7.90%
  • Voxtral, our run
    8.94%
  • MoulSot
    10.27%

Probe (a.k.a. Darija in the Wild)

  • Our 1.7B
    8.52%
  • Voxtral, published
    9.69%
  • Our 0.6B
    9.79%
  • Voxtral, our run
    12.11%
  • MoulSot
    14.93%

Voxtral’s published bars are from its own chart. Every other bar is our measurement. Our run used the released weights in 16-bit with the Transformers library, greedy decoding, and the default delay of 480 ms.

Our 1.7B model is slightly ahead of Voxtral’s published figures on both sets. Our 0.6B model is 1.2 points behind on Prism and 0.1 points behind on Probe, which is level. Both are well ahead of MoulSot. This is the same order as on our own test set.

Our own Voxtral run is higher

The dark bars are our own run of Voxtral Mini 4B Realtime Arabic. On Prism and Probe it gives CER of 8.94% and 12.11% and WER of 17.31% and 21.73%, against Voxtral’s published CER of 6.68% and 9.69%. The gap is most plausibly a preprocessing difference—text normalization, reference handling, or the decoding wrapper—rather than a difference in the model itself. We could not isolate it. The article therefore compares our models with the published figures, which are the more favourable ones for Voxtral.

Meaning error is robust to that difference. On Prism, our Voxtral run falls between our 1.7B model and our 0.6B model, which is where Voxtral’s published character error sits, even though our rerun’s character and word error are higher than both of our models. On Probe the three are closer: our 1.7B model is still ahead, and our 0.6B model is level with our Voxtral run. CER and WER move with the pipeline; the semantic ordering does not.

Meaning error on our Voxtral run (lower is better)
  • fihrIA-ASR
  • Voxtral, our run

Prism (a.k.a. ISMA)

  • Our 1.7B
    5.83%
  • Voxtral, our run
    7.95%
  • Our 0.6B
    8.88%

Probe (a.k.a. Darija in the Wild)

  • Our 1.7B
    4.65%
  • Voxtral, our run
    6.78%
  • Our 0.6B
    6.75%

Semantic error from the same judge prompt as the article, with one judge, GPT-5.6 Luna. These are our measurements, including our Voxtral run. Voxtral does not publish a meaning-error figure.

Character, word, and meaning error

Voxtral’s chart gives character error only. For the systems we ran, we also have word error and meaning error. Meaning error is the semantic error rate from the same judge prompt as the article, with one judge, GPT-5.6 Luna.

SystemPrism CERPrism WERPrism SERProbe CERProbe WERProbe SER
fihrIA-ASR 1.7B, 16-bit6.0512.975.838.5216.644.65
fihrIA-ASR 1.7B, 4-bit6.1913.496.139.1018.185.52
fihrIA-ASR 0.6B, 16-bit7.9016.338.889.7919.166.75
fihrIA-ASR 0.6B, 4-bit7.8716.239.059.7019.366.99
Voxtral Mini 4B, our rerun8.9417.317.9512.1121.736.78
MoulSot10.2722.2212.2614.9326.309.40
All values in percent, lower is better. All rows are our measurements. 16-bit and 4-bit are the precision of the decoder weights. The 0.6B rows are the 0.6B model with two small weight edits: one makes it stop on a clip without speech, and one breaks runaway repetition. MoulSot ran in 4-bit.

Meaning error gives the same picture. Our 1.7B model has the fewest meaning errors on both sets. Voxtral, in our run, and our 0.6B model are close: Voxtral is ahead on Prism (7.95% against 8.88%), and the two are level on Probe (6.78% and 6.75%). MoulSot is last on every measure.

What this adds to our own evaluation

Three things are different here: the clips, the references, and for Voxtral the people who ran the model. The order is the same. Our 1.7B model is slightly ahead of Voxtral, our 0.6B model is slightly behind or level, and both are clearly ahead of MoulSot. The difference in cost is not small: on one H100, our models run at 255× and 243× real time against 34× for Voxtral, which has 4 billion parameters.

Further in