How the model is built
Qwen3-ASR has two parts. The audio encoder turns the audio into a numerical representation. The decoder, a language model, takes this representation together with text, passed through a tokenizer (an optional context and the transcript written so far), and writes the transcript one token at a time. A token is a word or a piece of a word.
output: the next token
↑
Qwen3 language model (decoder)
reads the three inputs together and predicts the next token
↑
↑
Tokenizer
text → tokens → vectors
↑
context (optional):
names, terms
↑
↑
Audio encoder
sound → 12.5 vectors per second
↑
↑
↑
Tokenizer
text → tokens → vectors
↑
transcript so far:
language Darija<asr_text>واشنتادابا
↻ the predicted token (dashed) joins the transcript so far, and the model predicts the next one
Redrawn from Figure 2 (right) of the Qwen3-ASR technical report. Read from bottom to top. The language model receives three inputs side by side: an optional text context (for example names or terms likely to appear), the audio vectors from the encoder, and the transcript written so far. It predicts the next token, which is added to the transcript, and repeats until the end. The audio encoder (AuT) has about 300M parameters in the 1.7B model and 180M in the 0.6B model; the language model is Qwen3-1.7B or Qwen3-0.6B. We kept this architecture and replaced only the tokenizer, shown in blue: it converts text to tokens on the input side and defines the tokens the model can write on the output side.
Why the tokenizer sets the speed
The audio encoder runs once per clip, but the decoder runs once per token. So the number of tokens in a transcript largely determines how long transcription takes. Fragments have a second cost: the decoder has to learn to assemble Darija words out of pieces that carry no meaning on their own.
On our training transcripts, the original Qwen vocabulary (151,936 tokens) needs an average of 2.29 tokens per Darija word. Our Darija vocabulary (89,000 tokens) needs 1.32. That is 42% fewer decoder steps for the same sentence.
Original Qwen tokenizer (151,936 tokens): 18 pieces
Darija BPE (89,000 tokens): 9 pieces
On all our training transcripts, the Qwen vocabulary needs an average of 2.29 tokens for each word. The Darija vocabulary needs 1.32 tokens (42% fewer). Each token needs one decoder step.
Building the vocabulary
We trained a byte-level BPE tokenizer with 89,000 tokens on our Darija transcripts, plus French and English text so that borrowed words in the labels remain easy to write. The vocabulary is smaller than Qwen’s because it doesn’t need tokens for Chinese, Japanese, and the many other languages Qwen covers.
There is a trade-off. A few English words that were a single Qwen token now take two, while most French phrases with articles and elisions got cheaper:
Each row shows the number of tokens for one word in our transcripts: Qwen vocabulary, then Darija vocabulary. Green shows fewer tokens. Red shows more tokens.
Stage 1: initializing the new tokens
Each token is represented inside the model by a vector of numbers called an embedding, so a new vocabulary needs new embeddings. The 33,762 tokens that exist in both vocabularies (Arabic letters and word pieces, punctuation, digits, common Latin fragments) simply keep their trained Qwen embedding. The other 55,238 start from the average of all Qwen embeddings, a neutral point that doesn’t bias the model toward any particular word.
Stage 2: training the new embeddings
Next, we trained for 2,000 steps on Darija text, updating only the embeddings, at a learning rate of on four H100 GPUs. This step matters because all 55,238 new tokens start with the same value. If we trained the whole model straight away, they would push large, random changes into a decoder that was already well trained. Training them first moves them to useful positions.
Stage 3: training on speech
Finally, we trained the decoder on the 1.51 million clips. The audio encoder stayed fixed. It already handles speech, and the work is teaching the decoder to write Darija. Noise and music were mixed into the audio so the model is less tied to clean YouTube recordings. The learning rate was again , with 20 clips per GPU, for up to 5 epochs. We kept the checkpoint with the lowest validation loss: step 56,000 for the 0.6B model and step 50,000 for the 1.7B model.
Both sizes used this recipe. We also trained two control models with the same schedule, the same data, and the original Qwen tokenizer, so the comparison below changes only the vocabulary. At test time we decode with temperature 0 and Qwen3-ASR’s usual transcription prompt.
The comparison
0.6B
1.67× faster
1.7B
1.65× faster
Voxtral Mini 4B
Realtime Arabic · same GPU
The two models in each pair have the same architecture and the same training data. Only the tokenizer is different. We measured the transcription time on one H100 GPU (Transformers, batch size 32) for the 508 test clips (2.05 hours of audio). The time does not include model loading or file access. "255× real time" means that the model transcribes 255 seconds of audio in 1 second. SER on each tokenizer pair is Gemini 3 Flash, so the two models in a pair share one judge. Voxtral's SER is the mean of the four judges used in the main comparison. Voxtral Mini 4B Realtime Arabic was timed the same way. It is a different model, not a tokenizer variant.
For the 0.6B model, transcribing the 508 test clips on one H100 GPU drops from 48.1 seconds to 28.9 seconds. The speedup, about 1.65 times, is a little less than the 1.73 times that the token reduction alone would predict, because the audio encoder and batching overhead don’t get any faster.
Accuracy doesn’t change: at both sizes, WER moves by 0.3 points or less and SER by 0.1 points or less. The control models also show that model size has little effect on speed in this setup: the 0.6B and 1.7B controls run at 153× and 147× real time. Our 1.7B model has the same size as MoulSot, a 1.7B Qwen3-ASR fine-tune that keeps the original tokenizer, so the tokenizer accounts for all of its 1.65-times speedup over MoulSot.
The same H100 run also includes Voxtral Mini 4B Realtime Arabic. It finishes the 508 clips in 217.3 seconds, 34× real time, with WER 35.8%. Our 0.6B model is 7.5 times faster, and our 1.7B model is 7.1 times faster. Meaning errors for this comparison are the mean of the four judges: 7.6% and 6.4% for our models, and 8.4% for Voxtral.