What changes on disk
Only the decoder is quantized. The audio encoder stays in 16-bit in every run, for accuracy. The 16-bit format is BF16 and the 4-bit format is Q4_K_M, both from llama.cpp. You load two files: the decoder and the audio encoder.
| Decoder 16-bit | Decoder 4-bit | Audio encoder | Total 16-bit | Total 4-bit | |
|---|---|---|---|---|---|
| 1.7B | 3,188 MB | 1,000 MB | 642 MB | 3,830 MB | 1,642 MB |
| 0.6B | 1,068 MB | 342 MB | 379 MB | 1,447 MB | 721 MB |
Accuracy
| Prism CER | Prism WER | Prism SER | Probe CER | Probe WER | Probe SER | |
|---|---|---|---|---|---|---|
| 1.7B, 16-bit decoder | 6.05 | 12.97 | 5.83 | 8.52 | 16.64 | 4.65 |
| 1.7B, 4-bit decoder | 6.19 | 13.49 | 6.13 | 9.10 | 18.18 | 5.52 |
| 0.6B, 16-bit decoder | 7.90 | 16.33 | 8.88 | 9.79 | 19.16 | 6.75 |
| 0.6B, 4-bit decoder | 7.87 | 16.23 | 9.05 | 9.70 | 19.36 | 6.99 |
The 1.7B model loses a little at 4-bit. Its character error rises by 2.3% on Prism (6.05 to 6.19) and by 6.8% on Probe (8.52 to 9.10). Word error and meaning error move the same way on both sets.
The 0.6B model does not. Its character error is 7.90 at 16-bit and 7.87 at 4-bit on Prism, and 9.79 and 9.70 on Probe. Word error and meaning error differ by 0.24 points or less, in both directions.
How much of this is noise
Two things limit how far these differences can be read. Probe has only 109 clips, so one or two bad transcripts move its score. And the 16-bit runs were decoded on a CPU while the 4-bit runs were decoded on a GPU. The two do not compute exactly the same numbers: when we decoded the same 4-bit model on both, a few transcripts changed by a word or a punctuation mark. So the small differences of the 0.6B model should not be read as an effect of the precision. For the 1.7B model, every measure moves in the same direction, but we have not measured how much of the gap comes from the precision alone.
Speed
On an RTX 2060, a small desktop GPU, we transcribed the same 16 clips (214 seconds of audio) at both precisions. The 0.6B model runs at 40× real time at 4-bit against 32× at 16-bit. The 1.7B model runs at 27× against 19×.
Which one to use
For the 0.6B model, use 4-bit: on disk it is 721 MB instead of 1447 MB, it is faster, and we measured no clear loss. For the 1.7B model, 4-bit (1642 MB on disk against 3830 MB) is the practical choice on a small GPU or a laptop, and 16-bit is worth the space when accuracy matters most. The headline DarijaCS figures in the article use the 16-bit decoder; the 4-bit rows are in the table on the parent page.