Arabic speech recognition works for the news and breaks in the street.

Where the models stand

The Open Universal Arabic ASR Leaderboard scores open models on five test sets with one shared text normalisation.^[2]

Read MSA and broadcast speech now score below 20% for the best models. SADA, recorded from Saudi television in several dialects, is where every model loses. Whisper-large-v3 goes from 16.3% on MGB-2 to 56.0% on SADA.^[2]

The leaderboard’s best model is an Arabic-only Conformer from NVIDIA. Its 8.8% on Common Voice is far ahead of the multilingual models, but the paper does not say which datasets each model was trained on, so some of that lead may come from overlap with training data. One fine-tuned HuBERT model scores 8.0% on Common Voice and 67.8% on SADA, a gap that overlap would explain.^[2]

An earlier Interspeech study found the same gap with Whisper-large-v2. Zero-shot WER was 11.6% on FLEURS and 19.4% on Common Voice 11, both MSA. It rose to 34.7% on MGB-2, 43.5% on Egyptian MGB-3 and 83.0% on Moroccan MGB-5.^[3]

One language code, many spoken languages

Arabic speakers write in MSA but speak in a dialect, and the dialects differ in vocabulary, grammar and sounds. A Moroccan and an Iraqi can find each other hard to follow. ASR systems usually get one language code for all of them.

Talafha et al. collected about 48 hours of speech in eight dialects for Casablanca and tested four current models without fine-tuning.^[1] Even after normalising text, the best WER was 39.2% for Jordanian and 48.6% for Palestinian. For Moroccan, Algerian and Mauritanian Arabic, every model scored above 80%.^[1]

Dialect also changes the ranking. Whisper-large-v3 was best on Egyptian, Moroccan, Mauritanian and Yemeni, Whisper-large-v2 on Algerian, Palestinian and Emirati, and SeamlessM4T-v2 on Jordanian. MMS-1b-all was last everywhere.^[1]

The errors are often not random. Casablanca’s error analysis shows Whisper replacing dialect phrases with phonetically similar MSA words, or with an MSA translation of the phrase.^[1] A reader who knows Arabic may follow the output, but it is not a transcript.

Training on more MSA made this worse. Whisper-large-v2 fine-tuned on Common Voice, which is MSA, did worse than the untouched model on unseen dialects and accents in almost all of Talafha et al.’s 2023 tests.^[3] Fine-tuning on the right dialect helps. A Whisper model fine-tuned on Casablanca cut Algerian WER by 14 points compared with a model fine-tuned on Moroccan data.^[1]

FLEURS needs a caveat. Its Arabic set is MSA read by Egyptian speakers.^[3] A strong FLEURS score says little about spontaneous Egyptian speech, let alone Gulf or Maghrebi speech.

Writing without short vowels

Arabic script writes consonants and long vowels. Short vowels, doubled consonants and grammatical case endings are diacritics, and most people leave them out. A word like كتب can be read as “he wrote”, “books” or “it was written”, and readers resolve it from context.

ASR models mostly follow everyday text. Aldarmaki and Ghannam found that Whisper-medium, without fine-tuning, put a diacritic on 1.9 of every 100 letters when transcribing a fully diacritised Classical Arabic test set. The references had 84.6.^[5] That is fine for most products. It matters for language learning, religious text, text-to-speech pipelines, and any reference set written with diacritics.

Scoring with diacritics roughly doubles the error rate. A fine-tuned Whisper model trained on manually diacritised transcripts scored 6.5% WER with diacritics removed and 13.4% with them kept.^[5] Before comparing two numbers, check which kind they are.

Text normalisers can make this worse. Whisper’s default normaliser for non-English text replaces every Unicode combining mark with a space. Manohar et al. showed that this rule makes Indic-script results look better than they are.^[8] Arabic diacritics are combining marks too. We ran the rule on the diacritised word كَتَبَ (“he wrote”), and it came out as ك ت ب: one word became three. If the model writes diacritics and the reference does not, that single word now counts as one substitution and two insertions.

Hamza is a smaller version of the same problem. أ, إ, آ and ا are different codepoints, and writers use them inconsistently. The benchmarks above therefore strip diacritics and normalise hamza and madda forms before scoring.^{[1, 2, 3]} That is reasonable, but it means a published WER usually does not measure whether the model got these letters right.

No standard spelling for dialects

MSA has a spelling standard. The dialects do not. People write Egyptian or Moroccan Arabic the way it sounds to them, and two careful transcribers can write the same word differently.

Ali et al. measured how much this matters. They had five people transcribe the same Egyptian and North African speech and scored the same ASR output against each transcript. Against a single reference, WER averaged 71.4% for Egyptian and 80.1% for North African. When a word counted as correct if it matched any of the five references, WER fell to 39.7% and 45.9%.^[4] More than 30 points of the error came from spelling, not from recognition.

Most benchmarks still use one reference. Casablanca asked its annotators to write dialect words “in the manner they usually write in their daily lives”.^[1] That is a fair choice, but it means some of Casablanca’s high WERs are spelling disagreements a reader would never notice.

Numbers carry the same choice. Casablanca writes numbers as words, and its evaluation converts Eastern Arabic numerals (٣) to Western ones (3) before scoring.^[1] The leaderboard does the same conversion.^[2] If your references write digits and your model writes words, or the reverse, your WER will be high for reasons that have nothing to do with recognition.

English and French mid-sentence

Arabic speakers switch languages often: English in the Gulf and the Levant, French in North Africa, and both in professional speech.

In Casablanca, French code-switching is five times as common as English: 1,220 segments against 234, concentrated in the Algerian and Moroccan data.^[1] Whisper-large-v3 produced no code-switched words at all in any setting. Left to detect the language, or set to Arabic, it transliterated the foreign words into Arabic script. Set to English, it translated.^[1]

Mixat shows the same pattern for Emirati Arabic. It is 15 hours of podcast speech in which 36% of sentences mix Emirati Arabic and English.^[6] Whisper-medium scored 12.1% WER on the English-only segments, 121.8% on code-switched segments and 196.0% on Emirati-only segments. The authors found it was translating Emirati speech into MSA, often correctly, but that is the wrong task.^[6]

Purpose-built systems do better. On ArzEn, a corpus of code-switched Egyptian Arabic and English, Hamed et al.’s best system reached 30.6% WER by combining a hybrid and an end-to-end model.^[7] That is still far from the MSA results above.

Before you compare models on code-mixed Arabic, decide what a correct transcript of an English or French word looks like: Latin script or Arabic script. Then write your references that way.

Medical speech

Medical transcription is where errors cost the most. We could not find a public, peer-reviewed Arabic clinical speech test set with published ASR results.

A doctor-patient conversation in Arabic is spontaneous dialect speech, often with English or French medical terms, and every section above shows models doing worst on exactly that. Dialect speech alone pushed Whisper-large-v3 from 16.3% WER on broadcast speech to 56.0% on Saudi television.^[2]

The clinical pattern from other languages also applies. On accented clinical English from Africa, Afonja et al. found that models with low overall WER still made more errors on drug names, conditions and lab results, and that fine-tuning on clinical speech cut medical WER by 25 to 34% relative.^[9] An audit of clinical ASR in Indian hospitals found systems that did well on Indian English and failed on code-mixed and vernacular speech.^[10]

Anyone deploying ASR in an Arabic clinic should build a test set from their own patients first.

What helps

• Know which Arabic your users speak. Test on that dialect, not on MSA or FLEURS.^{[1, 3]}
• Fine-tune on in-dialect speech when you can. Fine-tuning Whisper on Casablanca cut Algerian WER by 14 points,^[1] and MMS fine-tuned on 668 hours of Saudi television reached 40.9% WER on SADA.^[11] Fine-tuning on MSA alone can make dialects worse.^[3]
• Fix the normalisation before you compare. Strip diacritics, unify hamza and madda forms, convert Eastern Arabic numerals, and don’t use a normaliser that turns diacritics into spaces.^{[1, 2, 8]}
• Report WER with and without diacritics if your use case needs them.^[5]
• For dialect speech, accept spelling variants. Multiple references, or a list of known variants, removes errors no reader would count.^[4]
• Write a policy for English and French words, then apply it to references and outputs alike.^{[1, 6]}
• Watch for translation. If a model’s output is fluent MSA when the speaker used dialect, it may be translating.^{[1, 6]}
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• A public Arabic clinical speech test set, in dialect, with consent, de-identification and clinician-checked references.
• Dialect-stratified results for commercial and newer open models on the same test set.
• A shared spelling convention for dialect transcripts, or a standard multi-reference scoring tool.
• Code-switching metrics that credit a correct transliteration of an English or French word.
• More conversational and telephone data for Gulf, Iraqi, Sudanese and Maghrebi Arabic.

We plan to cover several of these in QuantClarity’s own Arabic evaluations.

References

1. B. Talafha, K. Kadaoui, S. M. Magdy, et al.. Casablanca: Data and Models for Multidialectal Arabic Speech Recognition. EMNLP 2024.
2. Y. Wang, A. Alhmoud, M. Alqurishi. Open Universal Arabic ASR Leaderboard. arXiv preprint, 2024.
3. B. Talafha, A. Waheed, M. Abdul-Mageed. N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition. Interspeech 2023.
4. A. Ali, W. Magdy, P. Bell, S. Renals. Multi-reference WER for Evaluating ASR for Languages with No Orthographic Rules. IEEE ASRU 2015.
5. H. Aldarmaki, A. Ghannam. Diacritic Recognition Performance in Arabic ASR. Interspeech 2023.
6. M. Al Ali, H. Aldarmaki. Mixat: A Data Set of Bilingual Emirati-English Speech. SIGUL Workshop, 2024.
7. I. Hamed, P. Denisov, C.-Y. Li, M. Elmahdy, S. Abdennadher, N. T. Vu. Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech. Computer Speech & Language.
8. K. Manohar, L. G. Pillai, E. Sherly. What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations. EMNLP 2024.
9. T. Afonja, T. Olatunji, S. Ogun, N. A. Etori, A. Owodunni, M. Yekini. Performant ASR Models for Medical Entities in Accented Speech. Interspeech 2024.
10. S. Kumar et al.. ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages. arXiv preprint, 2025.
11. B. Gerazov, M. Politi, S. Bratières. Arabic ASR on the SADA Large-Scale Arabic Speech Corpus with Transformer-Based Models. arXiv preprint, 2025.