Hindi speech recognition looks better on benchmarks than it sounds in a clinic.

Where the models stand

Vistaar collected seven Hindi test sets and scored public and commercial systems on all of them.^[2]

IndicWhisper is Whisper fine-tuned on 10.7K hours of Indian speech, so it is not a zero-shot result. Every system does well on read speech. Gramvaani, recorded over phone lines from community radio callers, more than doubles the error rate for most of them.^[2]

Newer test sets show the same gap for zero-shot models. On Lahaja, a set of read and extempore Hindi from 132 speakers in 83 districts, Whisper-large-v3 scored 32.4% and MMS 34.4%. Google’s Chirp scored 22.3% and Azure 28.6%. A model trained on IndicVoices and other Indian data reached 14.3%.^[1] On the 2026 Vaani benchmark of spontaneous Hindi, Whisper-large-v3 scored 32.8% against a single reference, and the best system, Gemini-3.1-Pro, scored 19.0%.^[3]

Hindi as a second language

Hindi is spoken across India by people who learned it alongside another language. Their accent carries the sounds and rhythm of that first language. Most test sets do not record this.

Lahaja does. Of its 132 speakers, 122 have a first language other than Hindi, from 18 states.^[1] Error rates rise from Central and West India toward North-East and South India. For the best model in the paper, WER was 10.5% for native speakers of Urdu and Sindhi and 20.5% for native speakers of Assamese.^[1] The paper reports these regional trends in a figure, not a table, so we cannot give a full breakdown.

Speaker diversity in training data helped. A model trained on IndicVoices data from many Indian languages, with a wider range of speakers, beat the Hindi-only IndicVoices model by 2.7 points on Lahaja (16.7% against 19.4%).^[1]

If your users are in Chennai, Guwahati or Kolkata, a FLEURS or Common Voice score tells you little about how a model will do for them.

Names, payments and government services

Lahaja groups its prompts by topic, and the hardest ones are not the most technical. They are the ones full of names.^[1]

For Google Chirp, WER was 18.6% on read Wikipedia sentences and 16 to 20% on most extempore topics. It rose to 40.8% on digital payment requests, 52.3% on online government services and 63.2% on a task built around named entities.^[1] The IndicVoices-trained model stayed between 12.7 and 13.0% on payments and government services, but it too rose to 24.4% on the named-entity task.^[1]

Voice products often depend on exactly these words, such as a scheme, a bank or a district, and a model that does well on general sentences can still fail on them.

English words in Hindi sentences

Urban Hindi borrows English freely: “fever”, “BP”, “report”, “online”, “payment”. Transcribing these raises a question that most benchmarks answer differently. Should “fever” be written in Latin script or as फीवर?

The MUCS 2021 challenge measured how much English words cost. In its Hindi-English test set of technical lectures, 34.2% of words were English.^[4] Yet English words were behind 58 to 62% of all errors, depending on the system.^[4] The organisers also found that the reference transcripts themselves were inconsistent, sometimes writing the same English word in different scripts for the same audio. They added a transliterated WER (T-WER) that accepts an English word in either script. It lowered every system’s score, by 1.4 to 2.8 points.^[4]

The Vaani benchmark marks code-switched words in its references and accepts either script.^[3] Every system did worse on those words than on Hindi words, but the size of the penalty varied a lot. Substitution rates for one open Hindi model went from 12.8% to 28.3% on code-switched words. Google Chirp 3 went from 10.2% to 12.8%.^[3]

Before you compare models on Hindi with English in it, decide which script a correct English word uses, and score both scripts as correct if you accept both.

One spelling, several correct answers

Hindi spelling allows variation. A word borrowed from Persian can be written with or without a nukta (ज़रूरी or जरूरी). Nasal vowels can be written with a chandrabindu or a dot. English loanwords have no fixed Devanagari spelling. A careful transcriber might choose either form, and a scorer counts any mismatch as an error.

The Vaani benchmark had three people transcribe each clip independently. The three human transcripts differed from each other by 10.4 to 13.3% WER.^[3] When a model’s output was scored against all three references instead of one, average WER fell from 24.8% to 17.8%.^[3] Whisper-large-v3 went from 32.8% to 26.4%.^[3] The authors conclude that single-reference evaluation overstates errors.^[3]

An ICASSP 2026 study reached the same conclusion across Indian languages. Scoring against a set of valid spellings lowered error rates by 6.3 points on average and narrowed the gap between Gemini and NVIDIA’s Canary from 18.1 to 11.5 points.^[7]

The normaliser that hides vowel errors

Most Hindi WERs are computed after text normalisation. Whisper’s default normaliser for non-English text replaces every Unicode combining mark with a space.^[5] In Devanagari, that includes every matra and the virama.

Manohar et al. showed that this rule makes results for Indian languages look better than they are. On Hindi FLEURS, applying it lowered Whisper-small’s WER by 21.9 points.^[5] The published Whisper results for Hindi, 21.5% on FLEURS and 21.9% on Common Voice 9, were scored with this normaliser.^[6]

We ran the rule on a few Hindi words to see why. किताब (“book”) becomes क त ब: three fragments with no vowels. दिन (“day”) and दीन (“poor”) both become द न, so a model that writes the wrong word is scored as correct. So do पिता (“father”) and पीता (“drinks”). बुखार and बुख़ार, with and without a nukta, also end up equal, which is useful, but only because the rule throws away the nukta along with everything else.

When you read a Hindi WER, check which normaliser produced it. If it is Whisper’s default, vowel errors may not be counted at all.

Medical speech

Kumar et al. audited eight speech recognition systems on recorded consultations at a tertiary hospital for psychiatric and neurological care.^[8] The Hindi set had 58 doctor-patient interviews with a median length of 26 minutes, recorded on Android phones in busy wards and outpatient rooms.^[8]

The spread was wide. Gemini-2.5-pro scored 18.5% WER on Hindi. Sarvam’s Saarika-2.5 scored 39.0%. Whisper-large-v3 scored 71.7%, and Google STT 85.6%, mostly from deleted words.^[8] That is the same Whisper family that scores about 21% on read FLEURS.^[6]

Patients were harder to transcribe than doctors. For Sarvam on Hindi, patient speech had 56.2% WER and doctor speech 45.4%,^[8] and it is the patient’s words that a clinical record most needs.

The clinical pattern from other languages also applies. On accented clinical English from Africa, Afonja et al. found that models with low overall WER still made more errors on drug names, conditions and lab results, and that fine-tuning on clinical speech cut medical WER by 25 to 34% relative.^[9]

What helps

• Test on speakers like your users. Lahaja and Vaani are closer to real Hindi than FLEURS or Common Voice.^{[1, 3]}
• Fine-tune on diverse Indian speech. IndicWhisper cut Gramvaani telephone WER from 42.3% (Azure) to 26.8%,^[2] and speaker diversity in training data lowered Lahaja WER by 2.7 points.^[1]
• Don’t score Hindi with Whisper’s default normaliser. Use NFC normalisation and a Devanagari-aware rule set that keeps matras.^[5]
• Write a script policy for English words, and accept both scripts if you can (T-WER).^{[3, 4]}
• Allow spelling variants. Multiple references or a list of valid spellings can remove 5 to 7 points of error no reader would notice.^{[3, 7]}
• Build a test set of the names your product needs: banks, schemes, districts, drug names.^[1]
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• A public Hindi clinical speech test set with consent, de-identification and clinician-checked references.
• Region-by-region WER tables for current commercial and open models on the same accented test set.
• Studies of how models write Hindi numbers (digits or words) and how often that choice inflates WER.
• A shared standard for scoring English words in Hindi transcripts.
• Conversational and telephone results for current models, beyond Gramvaani.

We plan to cover several of these in QuantClarity’s own Hindi evaluations.

References

1. T. Javed, J. Nawale, S. Joshi, E. George, K. Bhogale, D. Mehendale, M. M. Khapra. LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems. Interspeech 2024.
2. K. S. Bhogale, S. Sundaresan, A. Raman, T. Javed, M. M. Khapra, P. Kumar. Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR. Interspeech 2023.
3. S. Pulikodan, A. Basu, S. Kumar, P. Bhat, P. Kumar J, N. Desai, P. K. Ghosh. Vaani Benchmark V1.0: An Inclusive Multimodal Benchmark Dataset for Hindi. arXiv preprint, 2026.
4. A. Diwan, R. Vaideeswaran, S. Shah, et al.. MUCS 2021: Multilingual and Code-Switching ASR Challenges for Low Resource Indian Languages. Interspeech 2021.
5. K. Manohar, L. G. Pillai, E. Sherly. What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations. EMNLP 2024.
6. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023.
7. K. S. Bhogale, T. Javed, G. S. John, D. Rathi, A. Padmanaban, N. Parasa, M. M. Khapra. Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages. ICASSP 2026.
8. S. Kumar et al.. ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages. arXiv preprint, 2025.
9. T. Afonja, T. Olatunji, S. Ogun, N. A. Etori, A. Owodunni, M. Yekini. Performant ASR Models for Medical Entities in Accented Speech. Interspeech 2024.
10. T. Javed, J. Nawale, E. George, et al.. IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages. Findings of ACL 2024.