Urdu speech recognition is harder than its benchmark scores suggest.

Where the models stand

Two recent studies test the same model families on Urdu without fine-tuning.

Whisper-large is large-v3 in the Common Voice and FLEURS results; the conversational study does not name the version. The conversational numbers also come from a different paper with its own text normalisation, so compare rankings across test sets rather than absolute values.

The ranking changes with the audio. SeamlessM4T-Large leads on read speech in both studies, while Whisper-large leads on conversational speech recorded over internet calls.^[2]

Small models fail badly. Whisper-tiny and Whisper-base score above 160% WER on the conversational set before fine-tuning.^[2] A WER over 100% means the model produced more errors than the reference has words, usually through hallucinated or repeated text.

Even the best score is far from what a clinical product needs. The Interspeech authors note that their best results remain well above the sub-10% WER expected for production use in domains such as healthcare, and that all of them are on read speech. Spontaneous, code-mixed speech would score worse.^[1]

Several ways to write each word

Urdu has no single spelling for many common words, and Unicode adds to the confusion.

Visually identical letters have different codepoints. Arabic Yeh and Farsi Yeh look the same but are different characters, and the same goes for Kaf and Keheh, Heh and Heh Goal, and several Hamza forms. A model that writes the “wrong” one is scored as making an error no reader would notice. Unifying these variants before scoring lowers Urdu WER by 0.4 points on Common Voice. The effect is larger for Pashto (3.1 points), and it costs nothing to fix.^[1]

Spelling variants cost more. Arif et al. found that “chahiye” (should, need) has several accepted spellings, which produced thousands of counted errors across models. “Hoga” (will be) appears as both ہوگا and ہو گا in the training data, 87 and 82 times, so fine-tuned models learned to output both and were penalised for it.^[2]

Text normalisers can hide errors or create them. Whisper’s default normaliser for non-English text replaces every Unicode combining mark with a space. Manohar et al. showed this makes Hindi and other Indic-script results look much better than they are, because it deletes vowel signs.^[6] Urdu diacritics such as zer and pesh are combining marks too. Run مُجھے through the same rule and you get م جھے: one word becomes two. Whether that raises or lowers the score depends on how the reference was written, so the same model can look better or worse because of a preprocessing choice.

Roman Urdu adds a second script. Many Pakistanis type Urdu in Latin letters, and some products need transcripts in that form. Naqvi et al. built a navigation system for TPL Maps that recognises street addresses in Roman Urdu mixed with English.^[4] Romanised text also blurs Urdu and Hindi: language identification systems routinely confuse romanised Hindi and romanised Urdu.^[5]

Urdu also leaks into its neighbours. Sindhi transcripts from current models contain 12 to 35% characters from other languages, mainly Urdu.^[1] For an Urdu product, this matters when your users actually speak Sindhi or Punjabi and the model is set to Urdu.

English in the middle of a sentence

Urdu speakers switch to English all the time, and ASR systems handle it poorly.

Khan et al. built a call-centre system for code-switched Urdu because regional calls mix the local language with English numbers and technical terms.^[3] It is one of the few systems built for this directly.

Across Urdu, Punjabi, Pashto and Sindhi, models drop 27 to 62% of the English words in the reference. They transliterate 9 to 54% into native script and keep only 1 to 11% in Latin script. MMS tends to delete them; SeamlessM4T tends to transliterate.^[1]

Some models translate. SeamlessM4T-Large turned the English word “research” into تحقیق, the Urdu word for research.^[2] The meaning survives, but the transcript no longer records what was said.

Scoring gets this wrong too. If a speaker says “scroll” and the model writes it in Urdu letters, a reader understands it, but WER counts it as an error.^[1] Before comparing models on code-mixed Urdu, decide what a correct transcript of an English word looks like, and write your references that way.

Who is speaking

Urdu is the first language of a minority of Pakistanis. Many speakers learn it alongside Punjabi, Pashto, Sindhi, Saraiki or Balochi, and their accent carries that first language.

Researchers at the Center for Language Engineering in Lahore classified five Urdu accents (Punjabi, Urdu, Pashto, Saraiki and Sindhi) from vowel features in 2014, and named accent among the leading causes of higher ASR error rates.^[11] The ARL read-speech set used by Arif et al. covers seven accent groups.^[2] A 2024 survey of Urdu ASR lists dialect and accent variation among the field’s main open problems.^[10]

What we could not find is a published breakdown of modern model WER by Urdu accent. The common benchmarks make this hard: Common Voice relies on sparse, self-reported accent labels, and FLEURS gives no dialect information at all.^{[1, 12]} A model’s FLEURS score tells you little about how it handles a Pashto speaker’s Urdu in Peshawar.

Medical speech

Medical transcription is where errors cost the most, and it has the least Urdu evidence. There is no public Urdu clinical speech benchmark that we know of. The closest resource is BioUNER, a 2026 dataset for named entity recognition in clinical Urdu text, with no audio.^[9]

Health is already a weak domain. In the Interspeech study, Urdu WER varied by 8 to 14 points across topic domains. Health was Whisper-medium’s worst domain at 46%, against 18% for SeamlessM4T-Large.^[1] Those are read sentences about health, not clinical conversation.

Rare words fail more often. Urdu words seen at most twice in training were recognised 30 to 45 points less accurately than common words.^[1] Drug names, test names and diagnoses are rare words by this measure.

Names are harder than other words. Across the four languages in the same study, named entities scored 5 to 20 points below other words in most configurations.^[1] Many drug names behave like names.

The clinical pattern holds in other languages. On accented clinical English from Africa, Afonja et al. found that models with low overall WER still made more errors on drug names, conditions and lab results, the errors most likely to harm a patient. Fine-tuning on clinical speech cut medical WER by 25 to 34% relative.^[7] An audit of clinical ASR in Indian hospitals found systems that did well on Indian English but failed on code-mixed and vernacular speech, with gaps between patients and clinicians.^[8]

Put together, a Pakistani doctor-patient conversation is close to a worst case: spontaneous, code-mixed, full of rare English terms, and often recorded in a noisy clinic. Anyone deploying ASR there should build a test set first.

Numbers and short replies

Numbers come out in different forms. Across the four languages on FLEURS, SeamlessM4T spells out 87 to 89% of numerals as words, MMS writes digits but gets fewer than a quarter right, and Whisper drops 21 to 27% of numerals entirely.^[1] Each model fails differently, so your evaluation needs a rule for دو versus “2”.

Short utterances are harder. Utterances of 1 to 5 words averaged about 76% WER, against 46% for those over 20 words.^[1] Part of this is arithmetic, since one wrong word in a one-word reply is 100% WER. But the replies that matter most in a voice flow, such as a yes, a digit or a name, are exactly these short ones.

What helps

• Set the language explicitly. Do not let the model guess between Urdu, Hindi and Punjabi.
• Normalise Perso-Arabic codepoints (Yeh, Kaf, Heh, Hamza, Alef) before scoring, and report raw and normalised WER together.^[1]
• Check what your text normaliser does to Urdu before trusting a score. Don’t reuse one built for English or Devanagari.^[6]
• Write a transcription policy for English words and numbers, and apply it to references and outputs alike.
• Score entities, numbers and medical terms separately from overall WER. Afonja et al.’s medical WER and entity recall are a good model.^[7]
• Test on audio like your users’. Read-speech rankings did not hold for conversational speech.^[2]
• Fine-tune when you can. Fine-tuning lowered every model’s Urdu WER in Arif et al.^[2] and improved Whisper on Pashto, Punjabi and Urdu in Sehar et al.^[13]
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• A public Urdu clinical speech test set, with consent, de-identification and clinician-checked references.
• WER broken down by accent and first language.
• A shared Urdu normalisation standard for ASR evaluation.
• Larger conversational and telephone test sets. The only public conversational set is 1.3 hours.^[2]
• Code-switching metrics that credit a correct transliteration.

We plan to cover several of these in QuantClarity’s own Urdu evaluations.

References

1. A. H. Azeemi, I. A. Qazi, M. Mustafa, A. A. Raza. Dissecting ASR Failures in Low-Resource South Asian Languages. Interspeech 2026.
2. S. Arif, S. Farid, A. J. Khan, M. Abbas, A. A. Raza, A. Athar. WER We Stand: Benchmarking Urdu ASR Models. COLING 2025.
3. M. D. Khan, R. Ali, A. Aziz. Code-Switched Urdu ASR for Noisy Telephonic Environment using Data Centric Approach with Hybrid HMM and CNN-TDNN. arXiv preprint, 2023.
4. S. M. R. Naqvi, M. A. Tahir, K. Javed, H. A. Khan, A. Raza, Z. Saeed. Code-Mixed Street Address Recognition and Accent Adaptation for Voice-Activated Navigation Services. IEEE Access 12, 2024.
5. E. Nielsen, C. Kirov, B. Roark. Distinguishing Romanized Hindi from Romanized Urdu. CAWL Workshop, ACL 2023.
6. K. Manohar, L. G. Pillai, E. Sherly. What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations. EMNLP 2024.
7. T. Afonja, T. Olatunji, S. Ogun, N. A. Etori, A. Owodunni, M. Yekini. Performant ASR Models for Medical Entities in Accented Speech. Interspeech 2024.
8. S. Kumar et al.. ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages. arXiv preprint, 2025.
9. W. Ali, A. Noor, S. Mahar, Alia, M. M. Younas. BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition. arXiv preprint, 2026.
10. M. Sharif, Z. Abbas, J. Yi, C. Liu. From Statistical Methods to Pre-Trained Models: A Survey on Automatic Speech Recognition for Resource Scarce Urdu Language. arXiv preprint, 2024.
11. Afsheen, S. Irtza, M. Farooq, S. Hussain. Accent Classification among Punjabi, Urdu, Pashto, Saraiki and Sindhi Accents of Urdu Language. Conference on Language and Technology, 2014.
12. M. Lau et al.. Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning. ACL 2025.
13. N. U. Sehar, A. Khalid, F. Adeeba, S. Hussain. Benchmarking Whisper for Low-Resource Speech Recognition: An N-Shot Evaluation on Pashto, Punjabi, and Urdu. CHiPSAL Workshop, COLING 2025.