Tamil speech recognition scores depend on how you count a Tamil word.

Where the models stand

Vistaar scored public and commercial systems on six Tamil test sets. All six are read speech.^[1]

IndicWhisper is best on five of the six sets. Azure is best on MUCS. No system goes below 17%, and most results sit between 25 and 40%.^[1]

Whisper’s own paper reports lower numbers for zero-shot Whisper-large-v2: 17.5% on Tamil FLEURS and 16.1% on Common Voice 9.^[2] Whisper’s paper scored them after its own text standardiser, and Vistaar scored its systems its own way, so the two sets of numbers are not one ranking.^{[1, 2]} Section 06 shows why the scoring step matters so much for Tamil.

Larger and more varied test sets are harder. On IndicVoices, a Canary model fine-tuned on Indian speech scored 27.0%, AI4Bharat’s IndicConformer 30.1%, Sarvam’s Saarika 44.1%, Gemini-2.5-Pro 44.4%, Azure 61.4% and Meta’s MMS 79.1%.^[3] A shared task on spontaneous speech from elderly Tamil speakers, recorded in hospitals, markets and shops, had a best WER of 39.5%.^[11]

Long words, one wrong suffix

Tamil builds words by stacking suffixes for case, number, tense and person. A single Tamil word can carry what English says in a phrase, so a word list never covers the language. Madhavaraj et al. built a Tamil system with a 182,771-word vocabulary from a 4.4-million-word text corpus.^[4] On their 65-hour test set, 10.73% of the words were still missing from the training vocabulary.^[4] Splitting words into hand-made morpheme units cut that to 1.68% and cut WER from 24.70% to 12.31%.^[4] Data-driven splitting with BPE or Morfessor cut WER by up to 6.24 points on the same data.^[5] Both studies come from one group, and neither has been repeated elsewhere that we could find.

WER is also harsh on long words. Get one suffix wrong and the whole word counts as an error. On Sri Lankan Tamil, Whisper-large-v3 had a WER of 80.7% but a character error rate of 42.5%.^[8] The gap suggests that many of its word errors were near misses.

Suffix errors still matter to the meaning. A Tamil case suffix can turn “to the doctor” into “from the doctor”. For a product, character error rate shows how close a model gets, and word error rate shows how often a field needs fixing.

Spoken Tamil has no single spelling

Tamil is diglossic. People write literary Tamil, which is standardised, and speak colloquial Tamil, which is not.^[6] The example above shows the gap: ரெண்டு for இரண்டு, இருக்கு for இருக்கிறது.

A transcriber of spoken Tamil has to choose between writing what was said and writing the textbook form. A model has to make the same choice. When the reference and the model choose differently, the scorer counts an error even if a Tamil reader would accept both.

Bhogale et al. measured how much this matters. They built a version of the IndicVoices test set with every valid spelling of each word, built with language experts, including colloquial forms and alternative uses of the pulli.^[3] Tamil needed 97,400 variants for 31,500 words, about three per word.^[3] Scored against those variants, error rates fell for every system: Canary from 27.0% to 17.2%, Gemini-2.5-Pro from 44.4% to 21.5%, Saarika from 44.1% to 23.5% and Azure from 61.4% to 43.6%.^[3]

Averaged over all systems, Tamil’s WER fell by 14.4 points, the largest gap the authors report, ahead of Malayalam at 13.9.^[3] The gap between Gemini and Canary on Tamil shrank from 17.4 points to 4.3.^[3] An older normalisation method missed several colloquial word forms.^[3]

If you compare Tamil models on a set like IndicVoices, much of the difference between them may come from spelling choices.

Dialects in India and Sri Lanka

Most Tamil test sets come from India. Two recent studies look at regional speech.

DravidianLangTech 2026 ran a shared task on four Tamil dialect groups: Northern, Southern, Western and Central.^[7] Teams had 9 hours and 22 minutes of training audio.^[7] Most teams fine-tuned Whisper models. The best system scored 51% WER, and the ten ranked entries ranged from 51 to 95%.^[7]

TaLK is the first speech benchmark for Sri Lankan Tamil. It has 1 hour and 33 minutes of spontaneous speech from 22 speakers, one per district.^[8] Zero-shot, Whisper-large-v3 scored 80.7% WER and MMS 84.5%.^[8] Whisper’s mean WER ranged from 67.2 to 90.3% across districts.^[8] Hatton, Moneragala and Badulla scored better for both models, which the authors link to speech there being closer to Indian Tamil.^[8]

TaLK is small, with one speaker per district, so a district’s score is one person’s score. Its authors say so.^[8] Still, if your users are in Jaffna or Batticaloa, a FLEURS score will not predict how a model does for them.

English words in Tamil sentences

Spoken Tamil borrows English words freely. A transcriber has to choose a script for each one, and so does a model.

Microsoft researchers trained and tested on Tamil-English speech, with a 19-hour code-switched test set.^[9] A model trained only on monolingual Tamil scored 50.1% WER on Tamil and 67.6% on Tamil-English.^[9] Adding code-switched training data brought the Tamil-English score to 55.9%, and their best model reached 53.2%.^[9]

The script choice causes errors of its own. In TaLK, the reference wrote English words in Tamil script, and Whisper-large-v3 often wrote them in Latin script instead.^[8] The spoken phrase இங்கிலீஷ் மீடியம் came out as “English Medium”, and லெக்சருக்கு (“to the lecture”) came out as “lecture கு”, an English word with a Tamil suffix.^[8] Whisper also tended to switch to English for numbers.^[8] Every one of these counts as an error against a Tamil-script reference.

Before you compare models on Tamil with English in it, decide which script a correct English word uses, and decide how to score a Tamil suffix on an English word.

The normaliser that hides vowel errors

Whisper’s default normaliser for non-English text replaces every Unicode combining mark with a space.^[10] In Tamil, that includes every vowel sign and the pulli. Manohar et al. found that the normalised Tamil text kept almost none of the original: its METEOR similarity to the original sentence was 0.^[10]

On Tamil FLEURS, Whisper-small scored 93.3% WER on raw text and 41.5 points lower after the normaliser.^[10] For a Whisper-small fine-tuned on Tamil, the normaliser removed 21.3 points.^[10] The words are cut into fragments, which adds words to the reference and makes each error count for less.^[10]

We ran the rule on a few Tamil words to see what it does. கால் (“leg”) and கோல் (“stick”) both become க ல, so a model that writes the wrong word is scored as correct. கிளி (“parrot”) and குளி (“bathe”) both become க ள. காய்ச்சல் (“fever”) becomes four fragments, க ய ச சல.

When you read a Tamil WER, check which normaliser produced it. If it is Whisper’s default, vowel errors may not be counted at all.

What helps

• Test on speech like your users’. Read-speech scores do not predict results on more varied speech. Azure scored 25 to 40% on read Tamil test sets and 61.4% on IndicVoices.^{[1, 3]}
• If your users are in Sri Lanka, test on Sri Lankan Tamil. TaLK is small, but it shows WERs far above Indian test sets.^[8]
• Write a transcription policy for spoken forms. Decide whether references use spoken or written Tamil, and accept both if you can. On Tamil, scoring against valid variants removed 18 to 23 points of error for the three commercial systems.^[3]
• Don’t score Tamil with Whisper’s default normaliser. Use one that keeps vowel signs and the pulli.^[10]
• Report character error rate next to WER. For long Tamil words, CER shows partial matches that WER hides.^[8]
• Use subword units if you train your own model. On one 65-hour test set, morpheme-like units cut WER from 24.7% to 12.3%.^[4]
• Write a script policy for English words and numbers, and train on code-switched speech if your users mix languages.^{[8, 9]}
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• A Tamil clinical speech test set with consent, de-identification and clinician-checked references. We could not find one. The closest evidence is from other languages: on accented clinical English, models with low overall WER still made more errors on drug names and conditions.^[12]
• A direct measure of how much WER changes when references are written in spoken versus literary Tamil.
• Tamil as spoken in Singapore, Malaysia and the diaspora.
• Telephone and call-centre Tamil results for current commercial models.
• Studies of names, numbers and addresses in Tamil speech recognition. TaLK notes number errors, but we could not find a study that measures them.
• A larger Sri Lankan Tamil test set, with more than one speaker per district.

We plan to cover several of these in QuantClarity’s own Tamil evaluations.

References

1. K. S. Bhogale, S. Sundaresan, A. Raman, T. Javed, M. M. Khapra, P. Kumar. Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR. Interspeech 2023.
2. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023.
3. K. S. Bhogale, T. Javed, G. S. John, D. Rathi, A. Padmanaban, N. Parasa, M. M. Khapra. Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages. ICASSP 2026.
4. Madhavaraj A, B. Pilar, A. G. Ramakrishnan. Knowledge-driven Subword Grammar Modeling for Automatic Speech Recognition in Tamil and Kannada. arXiv preprint, 2022.
5. Madhavaraj A, B. Pilar, A. G. Ramakrishnan. Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada. arXiv preprint, 2022.
6. M. Nanmalar, P. Vijayalakshmi, T. Nagarajan. Literary and Colloquial Tamil Dialect Identification. Circuits, Systems, and Signal Processing 41, 2022.
7. B. Bharathi, B. R. Chakravarthi, S. P. Muthusamy Chinnan, S. Saranya, S. Suhasini. Findings in Tamil Dialect Speech Recognition and Classification. DravidianLangTech 2026.
8. A. Thillainathan, N. Kanthakumar, N. Rasan, K. Sarveswaran. TaLK-Corpus: A Regionally Diverse Evaluation Set for Sri Lankan Tamil Speech. SPEAKABLE @ LREC 2026.
9. G. R. Madhumani, S. Shah, B. Abraham, V. Joshi, S. Sitaram. Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition. arXiv preprint, 2020.
10. K. Manohar, L. G. Pillai, E. Sherly. What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations. EMNLP 2024.
11. B. Bharathi, B. R. Chakravarthi, S. C. Navaneethakrishnan, N. Sripriya, A. Pandian, S. Valli. Findings of the Shared Task on Speech Recognition for Vulnerable Individuals in Tamil. LTEDI 2022.
12. T. Afonja, T. Olatunji, S. Ogun, N. A. Etori, A. Owodunni, M. Yekini. Performant ASR Models for Medical Entities in Accented Speech. Interspeech 2024.