Most Punjabi speakers write Shahmukhi. Punjabi speech benchmarks test Gurmukhi.

Where the models stand

The Interspeech 2026 study by Azeemi et al. tested five multilingual models on two public Gurmukhi test sets with one scoring pipeline.^[1]

SeamlessM4T-Large was best on both sets.^[1] Both Whisper models were near or above 100%, which in this study meant output in the wrong script.^[1] The best result, 22.1%, is more than twice the 10% target the authors name for production use in fields such as healthcare, and it is read speech.^[1]

The original Whisper paper found the same on FLEURS: every model size, from tiny to large-v2, scored between 101.5 and 103.6% WER on Punjabi.^[2]

Models trained on Indian data do better on Gurmukhi. The Vistaar benchmark from AI4Bharat compared systems across four Punjabi test sets in 2023.^[4] IndicWhisper, a Whisper-medium fine-tuned on 226 hours of Punjabi, scored 16.9% on Kathbath, 19.7% on its noisier Kathbath-Hard version, 22.4% on Common Voice and 23.1% on FLEURS.^[4] IndicWav2Vec scored between 15.1 and 34.8% on the same sets, and Google’s speech API between 25.5 and 44.0%.^[4] Azure’s speech API did not support Punjabi at the time.^[4] These numbers come from a different paper and setup than the chart above, so don’t rank them against it.

The newest numbers come from IndicVoices, a large Indian speech dataset. A 2026 arXiv preprint reports 10.7% WER on its Punjabi test set for a fine-tuned Canary model and 12.5% for AI4Bharat’s IndicConformer.^[7]

For Pakistani Punjabi, we found one published test. Sehar et al. built their own 4.2-hour Shahmukhi test set from 52 speakers in online vlogs, in the Majhi dialect.^[3] Whisper-large scored 54.7% WER zero-shot, and only after its Gurmukhi output was converted to Shahmukhi with a GPT prompt.^[3] Whisper-small, fine-tuned on 15 hours of Shahmukhi speech, reached 38.0%.^[3]

Two scripts, and the benchmarks use one

Punjabi in Pakistan and India is one spoken language with two written forms. Shahmukhi is a Perso-Arabic script, cursive, written right to left.^[5] Gurmukhi is a Brahmic script written left to right.^[5] Readers of one usually cannot read the other.^[5]

Most Punjabi speakers live in Pakistan.^[5] Yet every public Punjabi speech test set we found is in Gurmukhi: FLEURS, Common Voice, Kathbath and IndicVoices.^{[1, 4, 7]} Common Voice has no Shahmukhi Punjabi dataset at all.^[8] Sehar et al. had to build their own Shahmukhi data because they found no public dataset of Pakistani Punjabi with Shahmukhi transcripts.^[3]

When Sehar et al. asked Whisper for Punjabi, it wrote Gurmukhi for Pakistani speakers.^[3] Fine-tuning on 15 hours of Shahmukhi speech was what made it write Shahmukhi.^[3]

Converting between the scripts afterwards loses information. Shahmukhi leaves out most short vowels, and Gurmukhi spells them out.^[5] A rule-based Shahmukhi to Gurmukhi converter reached 98.95% word accuracy, but only on text where the missing vowel marks had first been added back by hand with dictionaries.^[5] Its author notes that leaving them out hurts accuracy a great deal.^[5] Real Shahmukhi text, and real transcripts, mostly leave them out.

For a product, decide which script each user reads before you choose a model. A score on FLEURS or Common Voice tells you about Gurmukhi output for Indian speakers. It says nothing about Shahmukhi for users in Pakistan.

Whisper writes Punjabi in Devanagari

On Common Voice, 95% of Whisper-medium’s Punjabi outputs were problematic.^[1] The authors’ fix, converting Devanagari to Gurmukhi, shows where much of it went: into Devanagari, the script of Hindi.^[1] Whisper-large-v3 failed in the same way at lower rates.^[1]

Hindi is the obvious source. Punjabi and Hindi share much vocabulary,^[1] and Whisper’s training data is very uneven between them. The Whisper paper lists 0.8 hours of Punjabi transcription data.^[2] Punjabi’s larger presence is in the translation data: 117 hours of Punjabi audio paired with English text.^[2] A model that has heard very little Punjabi transcribed in Gurmukhi falls back on a script it knows better.

The leakage also happens inside words. The Interspeech study counted hundreds of cases where models swapped a Gurmukhi character for a similar Devanagari one, such as ਰ for र, or the vowel sign ਾ for ा.^[1]

The sounds are often right even when the script is wrong. Azeemi et al. converted Whisper-medium’s Devanagari output to Gurmukhi with a rule-based transliterator.^[1] Its WER fell by 25.0 points, from 104.7 to 79.7%, and its CER by 62.5 points.^[1] That was the largest gain from transliteration in the study.^[1] Most of the speech had been heard correctly and written in the wrong alphabet. Even so, 79.7% is far behind SeamlessM4T’s 28.6% on the same set.^[1]

Check the script of every Punjabi transcript before you score it or show it to a user. WER over 100% is a strong hint that a model wrote the wrong script,^[1] but a check on the characters catches it directly.

Tone written with old letters

Punjabi is one of few Indo-European languages with lexical tone.^[6] A falling pitch can be the only difference between two words. An acoustic study of Indian Punjabi speakers gives pairs such as “sin” and “steam”, or “tune” and “paddy rice”, which differ only in a pitch fall on the first syllable.^[6]

The tone came from sounds the language lost. Punjabi once had voiced aspirated consonants, the sounds romanised as bh, dh and gh.^[6] At the start of a word they merged with plain voiceless consonants, and the breathy voice that lowered the pitch left a falling tone behind.^[6] The spelling did not change. “Steam” and “paddy rice” are still written with the letters for bh and dh, ਭ and ਧ.^[6] So a Gurmukhi letter like ਢ in ਢਿੱਡ now stands for a consonant sound plus a tone.

That makes spelling depend on pitch. To write the word correctly, a recogniser has to hear the pitch fall and pick the old aspirated letter. If it ignores pitch, the consonant it hears is plain. This is our reasoning about how recognisers work, not a measured result. We could not find a study that measures how often modern ASR models get tone words wrong in Punjabi.

The same study notes that analysis of Pakistani Punjabi speakers, in an unpublished manuscript by the same authors, found more variation between speakers and less frequent tone.^[6] A model trained mostly on Indian speakers may hear Pakistani speakers differently, and the other way round. We found no test of that.

Tone is not on every word. Only about 10% of Punjabi words come from the old aspirated consonants, so tonal words are relatively rare.^[6]

Valid spellings scored as errors

Some Punjabi words have more than one accepted spelling, and WER counts the one that doesn’t match the reference as an error. The Interspeech study found this in its Punjabi results: spellings with and without the gemination mark (ੱ, called addak), as in “vich” written with or without it, and variants in how aspirated sounds are written.^[1]

The addak doubles the consonant that follows it. ਵਿੱਚ and ਵਿਚ both mean “in” and both are in use. To a scorer they are different strings. But dropping the addak from every word before scoring would also be wrong, because in other words it is the only difference between two meanings: ਪਤਾ (“known”) and ਪੱਤਾ (“leaf”). This example is our own, not from a study.

How much do spelling variants add to Punjabi WER? A 2026 preprint from AI4Bharat built a Punjabi IndicVoices test set that lists the accepted variants for each word.^[7] Scored against it, the fine-tuned Canary model dropped from 10.7 to 7.9% WER, and IndicConformer from 12.5 to 9.8%.^[7] That is a gap of about 3 points between the score a model gets and the score it would get if valid spellings counted.^[7]

Unicode normalisation for Perso-Arabic scripts, which folds lookalike characters together, lowered Punjabi WER by 1.6 points on average in the Interspeech study.^[1] For Whisper on Punjabi it had almost no effect, because Whisper’s main errors were wrong scripts, not lookalike characters.^[1]

Thin data, few voices

Punjabi has very little open, checked speech. Common Voice 27, released in September 2026, holds 4.36 hours of recorded Punjabi, of which 2.44 hours are validated.^[8] It grew from 2.40 validated hours in Common Voice 22.^[8] All of it is in Gurmukhi.^[8]

The speakers are few and the women almost absent. 75 people have contributed.^[8] The release statistics label 63% of the data as male and about 1% as female; the rest has no gender label.^[8] A model trained or tested on Common Voice Punjabi has heard very few women’s voices.

Indian research datasets are much larger. The Vistaar training set collects 226 hours of Punjabi from several sources, most of it from Shrutilipi (94 hours) and Kathbath (124 hours).^[4] Hindi in the same collection has 2,150 hours, almost ten times as much.^[4] None of it is Shahmukhi.

We could not find Punjabi WER broken down by dialect for any modern model. The one Shahmukhi test set covers only Majhi, and its authors list that as a limitation.^[3] FLEURS gives no dialect information for any language,^{[1, 10]} and the Common Voice Punjabi release lists no accent labels.^[8]

Names, numbers and rare words

The Interspeech study measured these across Urdu, Punjabi, Pashto and Sindhi.^[1]

• Rare words: across the four languages, words seen at most twice in training were recognised at 20 to 62% accuracy, against 49 to 94% for common words.^[1] For Punjabi, word frequency was measured on only 800 training sentences.^[1] Drug names and place names are rare words by this measure.
• Names: named entities scored 5 to 20 points below other words in most configurations.^[1] Whisper’s entity accuracy on Punjabi was near zero, entirely because of wrong-script output.^[1]
• Short utterances: across the four languages, utterances of one to five words averaged about 76% WER, against 46% for utterances of 21 words or more.^[1] One wrong word in a one-word answer is 100% WER.
• Numbers: on FLEURS, SeamlessM4T spelled out 87 to 89% of numerals as words, MMS wrote digits but got under a quarter right, and Whisper dropped 21 to 27% of them.^[1]
• English words: models dropped 27 to 62% of English words in the reference, transliterated 9 to 54% and kept only 1 to 11% in Latin script.^[1] We could not find an ASR study of Punjabi-English code-switching.

What helps

• Decide the output script first. Gurmukhi benchmarks say nothing about Shahmukhi output for users in Pakistan.^{[1, 3]}
• Don’t use Whisper zero-shot for Punjabi. It scored above 100% WER at every size on FLEURS,^[2] and near or above 100% in a second study.^[1]
• For Gurmukhi, start from SeamlessM4T-Large or a model fine-tuned on Indian data. SeamlessM4T-Large was the best zero-shot model in one study, and fine-tuned Indian models reported lower WER on their own Indian test sets.^{[1, 4, 7]}
• For Shahmukhi, plan to fine-tune. Fifteen hours of Shahmukhi speech made Whisper-small write Shahmukhi and beat zero-shot Whisper-large.^[3]
• Check the script of every transcript, and flag Devanagari characters in Gurmukhi output.^[1]
• If you must use Whisper output, transliterate Devanagari to Gurmukhi. It cut Whisper-medium’s WER by 25 points and CER by 62.5 points without retraining.^[1]
• Score with a list of accepted spellings such as ਵਿੱਚ and ਵਿਚ, and report raw WER beside it.^{[1, 7]}
• Build your own test set with speakers from the regions and dialects your users come from, and include women.
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• A public Shahmukhi speech test set. The only published one is in-house.^[3]
• Spontaneous, conversational or telephone Punjabi from Pakistan.
• WER by dialect (Majhi, Doabi, Malwai, Puadhi, Pothwari and others), and for Pakistani against Indian speakers.
• How ASR models handle Punjabi tone words. We found acoustic studies of tone^[6] but no ASR evaluation of it.
• A Punjabi clinical speech test set with consent, de-identification and clinician-checked references. We could not find one. The closest evidence is from other languages: on accented clinical English, models with low overall WER still made more errors on drug names and conditions.^[11]
• Punjabi mixed with English, Urdu or Hindi in speech. We could not find an ASR study of Punjabi code-switching.
• Results from commercial speech APIs on a shared Shahmukhi test set.

We plan to cover several of these in QuantClarity’s own Punjabi evaluations.

References

1. A. H. Azeemi, I. A. Qazi, M. Mustafa, A. A. Raza. Dissecting ASR Failures in Low-Resource South Asian Languages. Interspeech 2026.
2. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023.
3. N. U. Sehar, A. Khalid, F. Adeeba, S. Hussain. Benchmarking Whisper for Low-Resource Speech Recognition: An N-Shot Evaluation on Pashto, Punjabi, and Urdu. CHiPSAL workshop at COLING 2025.
4. K. S. Bhogale, S. Sundaresan, A. Raman, T. Javed, M. M. Khapra, P. Kumar. Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR. Interspeech 2023.
5. M. G. A. Malik. Punjabi Machine Transliteration. COLING/ACL 2006.
6. J. Evans, W.-C. Yeh, R. Kulkarni. Acoustics of Tone in Indian Punjabi. Transactions of the Philological Society 116(3), 2018.
7. K. S. Bhogale, T. Javed, G. S. John, D. Rathi, A. Padmanaban, N. Parasa, M. M. Khapra. Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages. arXiv preprint, 2026.
8. Mozilla Common Voice. Dataset release statistics, Common Voice 22.0 (June 2025) and 27.0 (September 2026). GitHub, common-voice/cv-dataset.
9. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, A. Bapna. FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. IEEE SLT 2023.
10. M. Lau et al.. Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning. ACL 2025.
11. T. Afonja, T. Olatunji, S. Ogun, N. A. Etori, A. Owodunni, M. Yekini. Performant ASR Models for Medical Entities in Accented Speech. Interspeech 2024.