Every value on this page is a character error rate in percent, measured by us with the protocol of the parent page. Our two models are the 16-bit versions (fihrIA-ASR 1.7B, 16-bit and fihrIA-ASR 0.6B, 16-bit). The Voxtral column is our own run, which is higher than Voxtral’s published figures; Voxtral does not publish a breakdown.
By subject
| Clips | Our 1.7B | Our 0.6B | Voxtral, our rerun | MoulSot | |
|---|---|---|---|---|---|
| Tourism | 50 | 3.40 | 3.91 | 9.00 | 7.78 |
| Physics | 30 | 4.80 | 5.59 | 5.98 | 5.35 |
| Startups | 33 | 4.99 | 5.30 | 8.56 | 8.87 |
| Finance | 40 | 5.77 | 6.30 | 8.44 | 8.20 |
| History | 40 | 6.18 | 8.46 | 7.99 | 10.25 |
| Maths | 39 | 6.16 | 7.87 | 7.90 | 11.57 |
| Business | 40 | 6.63 | 7.87 | 7.90 | 7.88 |
| Life and earth sciences | 40 | 6.26 | 10.97 | 10.33 | 12.88 |
| Engineering | 45 | 9.72 | 13.74 | 12.01 | 16.52 |
Our 1.7B model has the lowest error of the four systems in every subject. Its easiest subject is tourism, at 3.40%, and its hardest is engineering, at 9.72%. Engineering is the hardest subject for every system. It is also one of the thinnest topics in our training data: about 9 of the 10,099 hours.
By background noise
| Clips | Our 1.7B | Our 0.6B | Voxtral, our rerun | MoulSot | |
|---|---|---|---|---|---|
| Clean | 178 | 4.42 | 5.84 | 7.45 | 8.55 |
| A little noise | 110 | 7.57 | 9.44 | 9.50 | 11.31 |
| Noisy | 36 | 11.13 | 17.12 | 15.42 | 17.68 |
Noise is the largest effect. From clean to noisy clips, character error more than doubles for every system: from 4.42% to 11.13% for our 1.7B model. Our 0.6B model loses the most. It is ahead of our Voxtral run on clean clips (5.84% against 7.45%) and behind it on noisy clips (17.12% against 15.42%).
By speaking pace
| Clips | Our 1.7B | Our 0.6B | Voxtral, our rerun | MoulSot | |
|---|---|---|---|---|---|
| Slow | 116 | 5.44 | 6.83 | 8.79 | 9.14 |
| Medium | 100 | 6.13 | 8.24 | 8.03 | 9.97 |
| Fast | 108 | 7.03 | 9.73 | 10.01 | 12.21 |
Fast speech is harder than slow speech for every system, but the effect is smaller than that of noise: 5.44% to 7.03% for our 1.7B model.
Probe, by clip type
| Clips | Our 1.7B | Our 0.6B | Voxtral, our rerun | MoulSot | |
|---|---|---|---|---|---|
| No label | 23 | 5.10 | 5.27 | 7.75 | 8.23 |
| Ordinary code-switching | 38 | 7.43 | 9.50 | 13.82 | 14.81 |
| Tricky code-switching | 11 | 4.74 | 4.86 | 4.05 | 9.02 |
| Technical code-switching | 9 | 14.09 | 15.13 | 19.56 | 26.78 |
| Ordinary and technical code-switching | 5 | 6.21 | 7.79 | 8.45 | 16.91 |
| Silence | 5 | 20.17 | 20.17 | 24.78 | 27.67 |
These groups are small, so read them as indications. Technical code-switching is the hardest kind of speech for our 1.7B model, at 14.09% on 9 clips.
On the 5 clips labelled silence, both fihrIA-ASR models match at 20.17% CER (27.78% WER), outperforming both Voxtral Mini 4B and MoulSot.