Speech models write Sindhi in Devanagari and in Urdu spelling.

Where the models stand

The Interspeech 2026 study by Azeemi et al. is the only peer-reviewed comparison of current multilingual models on Sindhi that we found.^[1] It tested five models on both public test sets with one scoring pipeline.

MMS-1B won both Sindhi pairs, the only language in the study where SeamlessM4T-Large did not win.^[1] Both Whisper models scored above 100% on both sets. A WER over 100% means the output has more errors than the reference has words, and in this study it meant the model wrote the wrong language.^[1]

The Common Voice column rests on 40 sentences, and the authors warn that this limits its statistical reliability.^[1] FLEURS, with 980 sentences, is the sturdier number. Even the best result, 23.5%, is more than twice the 10% target the authors name for production use in fields such as healthcare, and it is read speech.^[1]

The original Whisper paper reported the same pattern on FLEURS in 2022.^[2] Every Whisper size scored above 100% on Sindhi, and the larger models did worse: Whisper-tiny scored 105.8%, Whisper-large 177.9% and Whisper-large-v2 156.5%.^[2]

Fine-tuning changes the picture, but the only number we found is a community model card, not a paper. A Whisper-large-v2 model, first tuned on Hindi and then on the FLEURS Sindhi training set, reports 27.7% WER on the FLEURS test set.^[9] Treat it as an anecdote until someone tests it on a second set.

Whisper writes Sindhi in Devanagari

On Common Voice, every one of Whisper-medium’s Sindhi outputs was unusable: 63% were in Devanagari and the other 37% were garbage.^[1] Whisper-large-v3 failed in the same way at lower rates.^[1]

Devanagari is a plausible mistake. It is an official Sindhi script in India.^[3] But Sindhi text on the web is mostly Perso-Arabic, and Sindhi in Devanagari is a small fraction of it.^[3] A model that has heard little Sindhi and a lot of Hindi will write what it knows. The Whisper paper’s training data table backs this up. Its list of transcription data per language starts at 0.1 hours and does not include Sindhi; Sindhi appears only in the speech translation data, with 46 hours of Sindhi audio paired with English text.^[2]

The sounds are often right even when the script is wrong. Azeemi et al. converted Whisper’s Devanagari output to Sindhi Arabic script with a rule-based transliterator.^[1] Whisper-medium’s WER fell by 7.6 points and its CER by 33.4 points. Whisper-large-v3 gained more: 13.3 points of WER and 49.3 points of CER.^[1] The large drop in character errors shows that much of the speech was heard correctly and then written in the wrong alphabet.

The smaller drop in WER shows the limit. The two scripts do not map one to one, and written Sindhi usually leaves out the short vowels that Devanagari spells out.^[3] Even after conversion, both Whisper models stay far behind MMS.

For a product, a Whisper transcript of Sindhi may look like valid text and be in the wrong script for most of your users in Pakistan. A check on the script of each transcript catches it; WER alone only tells you the score is bad.

Urdu leaks into Sindhi output

Models that do write Perso-Arabic script still mix in another language. 12 to 35% of the characters in Sindhi outputs came from other languages, mostly Urdu.^[1] Urdu outputs showed almost none.^[1] The authors tie this to training data: the less Sindhi a model has seen, the more it borrows from its better-resourced neighbour.^[1]

Urdu and Sindhi share most of the Arabic alphabet but not all of it, and the differences fall on common words. Sindhi writes the “k” in ڊاڪٽر with ڪ (U+06AA, Swash Kaf), and Urdu uses ک (U+06A9, Keheh).^[3] Sindhi’s retroflex ڊ and ٽ are different codepoints from Urdu’s ڈ and ٹ. So an Urdu-spelled ڈاکٹر differs from the Sindhi word in three of its five letters, and WER counts it as one wrong word.

The scorer cannot tell this apart from a real error, and normalisation helps little. Folding lookalike Yeh, Kaf, Heh, Hamza and Alef variants before scoring lowered Sindhi WER by only 0.8 points on average, against 3.1 for Pashto.^[1] Sindhi’s errors are mostly wrong scripts and wrong letters, which these rules do not touch.^[1]

Folding the Urdu letters into the Sindhi ones would be a mistake. In Sindhi, ک is a separate letter for the aspirated “kh”. It is the first letter of کان (“from”) in our example sentence. A rule that merged them would score a real error as correct. We checked that Unicode NFC and NFKC normalisation leave ڪ and ک as different characters. These are our own checks of the Unicode rules, not a published result.

Before you compare Sindhi scores, check which characters the scorer folded together, and count Urdu letters in the output separately from recognition errors.

Recorded, but not checked

Common Voice 27, released in September 2026, holds 73.38 hours of recorded Sindhi.^[4] Only 0.36 hours of it are validated, meaning that other volunteers listened and confirmed the recording matches the sentence.^[4] In the same release, Pashto has 3,530 validated hours and Urdu 79.85.^[4]

The recorded set grew from 29.13 hours in Common Voice 22 to 73.38 hours in Common Voice 27, while the validated share stayed at 0.36 hours.^[4] This is why the Common Voice test set used by Azeemi et al. had 40 sentences and why only 271 Sindhi training sentences were available to measure word frequency.^[1]

The speakers are also few and narrow. 35 people have contributed to Sindhi Common Voice.^[4] The release statistics put 75% of the data with speakers in their forties and 24% in their thirties.^[4] 80% carries no gender label, about 20% is labelled male, and none is labelled female.^[4] A model trained or tested on this data has heard very few Sindhi voices, and none labelled as women.

Validating the Sindhi clips already recorded, with no new recording, would turn tens of hours of audio into usable training and test data.

Dialects and who is in the data

The usual account names six Sindhi dialects.^[5] The Middle dialect (Vicholi) borders all four other dialects of Sindh and is the basis of written, standard Sindhi; other dialects appear in writing mostly in poetry and stories.^[5] A 2010 acoustic study recorded speakers from all five dialects in Sindh and found that vowels shift between them, and that some words change their vowel sounds: Lari speakers drop a final diphthong that Middle and Utradi speakers keep.^[5] Sindhi researchers have named accent variation as one of the main obstacles to Sindhi speech recognition.^[8]

We could not find WER broken down by Sindhi dialect. The public data does not allow it. The Sindhi release statistics list no accent labels,^[4] and FLEURS gives no dialect information for any language.^{[1, 7]} With 35 contributors, the Common Voice set cannot cover six dialects in any case.^[4]

The same gap exists across the border. Sindhi speakers in India write in Devanagari as well as Perso-Arabic.^[3] We could not find a public Sindhi speech test set in Devanagari, or any test of how models handle Sindhi speakers from India or the Kutch region.

Names, numbers and rare words

The Interspeech study measured these across Urdu, Punjabi, Pashto and Sindhi.^[1]

• Rare words: across the four languages, words seen at most twice in training were recognised at 20 to 62% accuracy, against 49 to 94% for common words.^[1] The gap persisted for Sindhi on FLEURS even with only 271 training sentences to count from.^[1] Drug names and place names are rare words by this measure.
• Names: named entities scored 5 to 20 points below other words in most configurations.^[1] Whisper’s entity accuracy on Sindhi was near zero, entirely because of wrong-script output.^[1]
• Short utterances: across the four languages, utterances of one to five words averaged about 76% WER, against 46% for utterances of 21 words or more.^[1] One wrong word in a one-word answer is 100% WER. Short answers such as yes, no, a number or a name are what many voice products depend on.
• Numbers: on FLEURS, SeamlessM4T spelled out 87 to 89% of numerals as words, MMS wrote digits but got under a quarter right, and Whisper dropped 21 to 27% of them.^[1]
• English words: models dropped 27 to 62% of English words in the reference, transliterated 9 to 54% and kept only 1 to 11% in Latin script.^[1]

What helps

• Don’t use Whisper zero-shot for Sindhi. It scored above 100% WER at every size in two separate studies.^{[1, 2]}
• Start from MMS-1B. It had the best zero-shot Sindhi WER on both public test sets.^[1]
• Check the script of every transcript. Flag Devanagari output, and count Urdu letters such as ڈ and ٹ in place of Sindhi ڊ and ٽ, and ک where Sindhi writes ڪ.^[1]
• If you must use Whisper output, transliterate Devanagari to Sindhi script. It cut CER by 33 to 49 points without retraining.^[1]
• Normalise before scoring, but don’t fold ڪ into ک. Report raw and normalised WER together.^[1]
• Build your own test set. The public Common Voice test set has 40 sentences; record speakers from the dialects and regions your users come from.
• Write a policy for numbers and English words, and apply it to references and outputs alike.^[1]
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• Spontaneous, conversational or telephone Sindhi. Every public number is read speech.^[1]
• A peer-reviewed fine-tuning study for Sindhi. The only fine-tuned number we found is a model card.^[9]
• WER by dialect, including Kachhi and speakers from India. No public test set labels it.^{[4, 7]}
• A Sindhi speech test set in Devanagari.
• A Sindhi clinical speech test set with consent, de-identification and clinician-checked references. We could not find one. The closest evidence is from other languages: on accented clinical English, models with low overall WER still made more errors on drug names and conditions.^[10]
• Sindhi mixed with Urdu or English in speech. We could not find an ASR study of Sindhi code-switching.
• Results from commercial speech APIs on a shared Sindhi test set.

We plan to cover several of these in QuantClarity’s own Sindhi evaluations.

References

1. A. H. Azeemi, I. A. Qazi, M. Mustafa, A. A. Raza. Dissecting ASR Failures in Low-Resource South Asian Languages. Interspeech 2026.
2. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023.
3. R. Motlani. Developing language technology tools and resources for a resource-poor language: Sindhi. NAACL Student Research Workshop 2016.
4. Mozilla Common Voice. Dataset release statistics, Common Voice 22.0 (June 2025) and 27.0 (September 2026). GitHub, common-voice/cv-dataset.
5. A. Keerio. Acoustic Analysis of Sindhi Speech: A Pre-cursor for an ASR System. DPhil thesis, University of Sussex, 2010.
6. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, A. Bapna. FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. IEEE SLT 2023.
7. M. Lau et al.. Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning. ACL 2025.
8. T. Qureshi, D. N. Hakro, F. H. Khoso, K. U. R. Khoumbati, R. Takharwal, M. Hameed, G. N. Rajper. Issues and Challenges in Sindhi Speech Recognition Systems. Sindh University Research Journal (Science Series) 50(4), 2018.
9. steja. whisper-large-sindhi. Community model card on Hugging Face, labelled anecdote.
10. T. Afonja, T. Olatunji, S. Ogun, N. A. Etori, A. Owodunni, M. Yekini. Performant ASR Models for Medical Entities in Accented Speech. Interspeech 2024.