Telugu speech recognition is only as good as the Telugu it was trained on.

Where the models stand

Vistaar scored public and commercial systems on five Telugu test sets. All five are read speech.^[1]

No system wins everywhere. Azure is best on Kathbath, Kathbath-Hard and MUCS, IndicWhisper on FLEURS, and IndicWav2Vec on IndicTTS.^[1] IndicTTS, studio recordings of read sentences, is the hardest Telugu set for all four systems. Azure scores 61.1% there and 16.1% on MUCS.^[1]

On IndicVoices, a larger and more varied test set, a Canary model fine-tuned on Indian speech scored 22.9%, AI4Bharat’s IndicConformer 25.5%, Gemini-2.5-Pro 33.4%, Sarvam’s Saarika 33.5%, Azure 52.4% and Meta’s MMS 68.5%.^[3] The MUCS 2021 challenge baseline, a conventional Kaldi system trained on 40 hours of Telugu, scored 28.7% on its Telugu test set.^[4]

A bigger Whisper is not a better Telugu Whisper

Whisper’s own paper reports its Telugu FLEURS results by model size.^[2] Tiny scored 105.1%, base 105.2%, small 98.2%, medium 82.8%, large 100.6% and large-v2 99.0%.^[2] A WER above 100% means the model added more wrong words than the reference had. For comparison, large-v2 scored 17.5% on Tamil FLEURS in the same table.^[2]

The authors list Telugu among the largest outliers: languages that do worse than their amount of training data predicts.^[2] They suggest three possible causes: Telugu is distant from the European languages that dominate the training data, Whisper’s byte-level tokenizer may suit its script poorly, and the training data may be of lower quality.^[2] The paper does not test which one it is.

Fine-tuning closes most of the gap. Whisper fine-tuned on Vistaar’s Indian data scored 25.4% on the same FLEURS test.^[1] In a 2026 preprint, Whisper-medium fine-tuned on Kathbath scored 23.3% on Kathbath’s Telugu test.^[8]

If you are choosing a model for Telugu, a general multilingual model’s overall score tells you little. Test it on Telugu, and expect a model trained on Indian Telugu to do better.

Three dialects, one language model

Telugu has three broad regional dialects: Coastal Andhra, Telangana and Rayalaseema.^[5] Yadavalli et al. describe Telangana speech as shaped by Urdu, with words such as దవాఖానా (“hospital”), Coastal Andhra by Sanskrit and English, and Rayalaseema by Tamil and Kannada.^[5]

The same group trained Telugu systems on about 1,200 hours of crowd-sourced spontaneous speech from all three regions, with about 8 hours of test speech per dialect.^[5] Separate systems for each dialect scored 34.3% WER on Coastal Andhra, 25.4% on Telangana and 24.2% on Rayalaseema.^[5] One shared system with a dialect label added to its output scored 30.8%, 27.0% and 20.8%.^[5]

The language model matters too. In a second study, the group kept a dialect-specific acoustic model and swapped the language model.^[6] Telangana speech scored 25.4% with a Telangana language model, 40.1% with a Coastal Andhra one and 40.9% with a Rayalaseema one.^[6] Averaged over the three dialects, WER rose from 28.0% with matched language models to 34.9% with mismatched ones.^[6] For Telangana, using no language model at all (27.9%) beat using the wrong one.^[6] Rayalaseema was the exception: its WER rose by only about one point, which the authors link to its overlap with both other dialects.^[6]

The authors note that most Telugu text from outside sources is in the standardised dialect.^[6] A model whose language model learned from news and Wikipedia is in the mismatched case for many speakers. Both studies come from one lab and one corpus, and we could not find an independent replication.

English inside Telugu

Spoken Telugu borrows English words freely. In a Microsoft study of Tamil, Telugu and Gujarati speech mixed with English, Telugu-English had the highest code-mixing index of the three, on both training and test data.^[7]

The study had 243 hours of Telugu-English training speech and a 28-hour Telugu-English test set.^[7] A model trained only on monolingual Telugu scored 46.9% WER on Telugu and 59.9% on Telugu-English.^[7] Training on both kinds of speech brought the Telugu-English score to 40.8%.^[7] Their best model scored 39.2% on Telugu-English and 43.1% on monolingual Telugu, so with mixed training data the mixed test set ended up the easier one.^[7]

These systems were trained on the study’s own data. We could not find a study that tests current commercial systems or Whisper on Telugu-English speech, or one that measures which script a model uses for English words in Telugu.

Before you compare models on Telugu with English in it, decide which script a correct English word uses, and check that your test set has as much mixing as your users’ speech.

Long words and many valid spellings

Telugu builds words by adding suffixes, so the number of distinct words grows fast. On Kathbath’s Telugu data, a 2026 preprint counted 134,100 unique words, which morpheme splitting reduced to 31,340 units.^[8] Most of the fine-tuned Whisper’s errors were in words it had seen in training: 16.8 of its 23.3 WER points.^[8] Scoring on morpheme units instead of words lowered WER from 23.3% to 18.7%.^[8] Part of that drop comes from changing the unit being counted, so the two numbers do not measure the same thing.

The Yadavalli group report character error rate as well, because they find WER less reliable for an agglutinative language like Telugu.^[6] In their tables, CER sits between 7.6 and 11.6% where WER sits between 24.2 and 34.3% for the same matched systems.^[6]

Spelling adds more noise. Bhogale et al. rebuilt the IndicVoices test set with every valid spelling of each word, built with language experts.^[3] Telugu needed 92,700 variants for 29,100 words, about 3.2 per word, the highest ratio in their table by our count.^[3] Scored against those variants, error rates fell for every system: Canary from 22.9% to 13.7%, IndicConformer from 25.5% to 16.5%, Gemini-2.5-Pro from 33.4% to 19.4%, Saarika from 33.5% to 20.6%, Azure from 52.4% to 40.5% and MMS from 68.5% to 62.0%.^[3]

By our calculation from their table, the average drop for Telugu is 10.6 points, fourth largest of the eleven languages shown, after Tamil, Malayalam and Odia.^[3] The authors also found that Telugu had one of the largest gaps between this spelling-aware score and what human judges counted as errors, which they attribute to audio that is ambiguous without more context.^[3]

If you compare Telugu models on one reference spelling, part of the difference between them may come from spelling.

The normaliser that hides vowel errors

Whisper’s default normaliser for non-English text replaces every Unicode combining mark with a space.^[9] Manohar et al. showed that this cuts Tamil and Malayalam words into fragments and lowers their WER by tens of points.^[9] They did not test Telugu, but Telugu vowel signs, the pollu and the anusvara are combining marks too.

We ran the rule on a few Telugu words to see what it does. కాలు (“leg”) and కీలు (“joint”) both become క ల. మందు (“medicine”) and ముందు (“before”) both become మ ద. జ్వరం (“fever”) becomes జ వర. In each pair, a model that writes the wrong word is scored as correct.

When you read a Telugu WER, check which normaliser produced it. If it is Whisper’s default, vowel errors may not be counted at all.

What helps

• Test on speech like your users’. Scores swing widely even between read-speech test sets: Azure scored 16.1% on MUCS and 61.1% on IndicTTS.^[1]
• Don’t pick a model for Telugu from its overall multilingual score. Zero-shot Whisper-large-v2 scores 99.0% on Telugu FLEURS, and a fine-tuned Whisper 25.4%.^{[1, 2]}
• Know where your users are. If they are in Telangana, test on Telangana speech and use a language model trained on Telangana text. In one study, a mismatched language model added about 15 points of WER on Telangana speech.^[6]
• Train on code-switched speech if your users mix in English. In one study, it cut Telugu-English WER from 59.9% to 40.8%.^[7]
• Accept valid spelling variants when you score. On Telugu, it removed 9 to 14 points of error for the five best systems.^[3]
• Don’t score Telugu with Whisper’s default normaliser. Use one that keeps vowel signs, the pollu and the anusvara.^[9]
• Report character error rate next to WER. For long Telugu words, CER shows partial matches that WER hides.^[6]
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• A Telugu clinical speech test set with consent, de-identification and clinician-checked references. We could not find one. The closest evidence is from Kannada, a neighbouring Dravidian language: on psychiatric hospital recordings, the best system scored 35.0% WER.^[10]
• Telugu-English results for current commercial models and Whisper, and a study of which script they use for English words.
• An independent dialect test set. The dialect results above come from one lab and one corpus.
• Telephone and call-centre Telugu.
• Studies of names, numbers and addresses in Telugu speech recognition. We could not find one.
• A published Telugu WER with and without Whisper’s default normaliser.

We plan to cover several of these in QuantClarity’s own Telugu evaluations.

References

1. K. S. Bhogale, S. Sundaresan, A. Raman, T. Javed, M. M. Khapra, P. Kumar. Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR. Interspeech 2023.
2. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023.
3. K. S. Bhogale, T. Javed, G. S. John, D. Rathi, A. Padmanaban, N. Parasa, M. M. Khapra. Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages. ICASSP 2026.
4. A. Diwan, R. Vaideeswaran, S. Shah, et al.. Multilingual and Code-Switching ASR Challenges for Low Resource Indian Languages. Interspeech 2021.
5. A. Yadavalli, G. S. Mirishkar, A. K. Vuppala. Multi-Task End-to-End Model for Telugu Dialect and Speech Recognition. Interspeech 2022.
6. A. Yadavalli, G. S. Mirishkar, A. K. Vuppala. Exploring the Effect of Dialect Mismatched Language Models in Telugu Automatic Speech Recognition. NAACL 2022 Student Research Workshop.
7. G. R. Madhumani, S. Shah, B. Abraham, V. Joshi, S. Sitaram. Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition. arXiv preprint, 2020.
8. C. V. Kumar, K. Tripathi, P. Wasnik. Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languages. arXiv preprint, 2026.
9. K. Manohar, L. G. Pillai, E. Sherly. What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations. EMNLP 2024.
10. S. Kumar, P. Shivaprakash, A. Manoharan, A. Kurariya, D. Mukherjee, L. Shukla, A. Mukherjee, P. Chand, P. Murthy. ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages. arXiv preprint, 2025.