Most speech models cannot write Pashto in Pashto.

Where the models stand

A 2026 preprint by Hanif Rahman tested ten models on the same two Pashto test sets with one scoring pipeline.^[2] Both sets are read speech. The author also maintains several of the fine-tuned models in the paper and contributed much of the Pashto Common Voice data, and says so.^[2]

MMS, SeamlessM4T and OmniASR land between 40 and 62%. Every Whisper model is near or above 90%, and every Whisper model goes above 100% on Common Voice.^[2] Whisper-medium reached 461.2% on Common Voice.^[2] A WER over 100% means the output has more errors than the reference has words, usually from repeated or invented text.

The Interspeech 2026 study by Azeemi et al., which used Common Voice 22, found the same order. SeamlessM4T-Large scored 42.0% on Common Voice and 43.8% on FLEURS. Whisper-large-v3 scored 90.4% and 89.2%.^[1]

The only results beyond read prompts come from broadcast news. On 4.8 hours of Afghan Pashto news, zero-shot Whisper-large scored 85.6% and Whisper-small 98.4%.^[3] Whisper-small fine-tuned on 1 hour of news reached 53.1%, and on 15 hours 34.1%.^[3] That study converted some output spellings with a GPT prompt before scoring, which another author notes makes it hard to repeat.^{[2, 3]}

Whisper writes Pashto as another language

Rahman checked which script each model wrote on FLEURS.^[2] MMS-1B wrote Pashto script in 95.1% of sentences, SeamlessM4T in 97.1% and OmniASR in 93.8%. No Whisper model went above 0.8%.^[2] Whisper-large-v3 wrote 99.2% of its output in Arabic, Dari or Urdu script, even with the language forced to Pashto.^[2] Smaller Whisper models wrote much of their output in Latin letters, and Whisper-medium wrote 59.6% of sentences in no clear script at all.^[2]

The wrong-script output keeps the sounds but swaps the letters. In one FLEURS sentence with six Pashto-only characters, Whisper-large-v3 wrote none of them: ټ became ت, ۍ became ی, and ښ became خ.^[2]

The Interspeech study found the same failure on Common Voice. 80% of Whisper-medium’s Pashto outputs were unusable: 22% in the wrong script and 58% garbage, including Khmer characters.^[1] Removing those outputs alone dropped its WER from 165.3% to about 97%.^[1] The broadcast study saw Whisper-small and medium switch languages mid-transcript even when told to use Pashto, and saw Whisper switch between Northern and Southern spelling conventions.^[3]

Whisper-medium scored worse than Whisper-small on both sets, and its output was full of loops: the same short word repeated to the length limit.^[2]

For a product, this means a Whisper transcript of Pashto can look like a transcript and not be Pashto. A Pashto reader may follow some of it, but a search index or a downstream model will not.

Letters the keyboard leaves out

Pashto’s eight extra consonant letters are not on standard Arabic or Persian keyboards.^[4] People who type without a Pashto layout swap in the nearest Arabic letter, or drop the letter if nothing looks close.^[4] The builders of Pashto Common Voice found ښ, ږ, څ and ځ to be the four letters dropped most often, and collected extra sentences containing them.^[4]

Text typed this way trains models to make the same swaps, and a reference typed this way marks a correct model as wrong.

The sounds behind these letters are also the hardest for models. In a fine-tuned W2V-BERT model scored at 35.4% WER on FLEURS, sentences with ږ scored 1.6 points worse than average, ښ 1.0 point worse and the retroflex ټ and ډ 0.7 points worse.^[2] Letters shared with Arabic scored at or below average.^[2] A separate study of fine-tuned Whisper models found the most common errors were masculine and feminine word endings swapped, and substitutions involving څ.^[6] These endings carry grammar, so a wrong one can change a word’s gender or number.

Lookalike letters and the scorer

Some Arabic-script letters have more than one Unicode codepoint and look the same in many positions. The Interspeech study counted 11,547 cases in its Pashto outputs where the only difference between reference and model was Arabic Yeh (U+064A) against Farsi Yeh (U+06CC).^[1] Folding these and similar variants for Kaf, Heh, Hamza and Alef before scoring lowered Pashto WER by 3.1 points on average, more than for Punjabi (1.6), Sindhi (0.8) or Urdu (0.4).^[1]

Rahman’s pipeline uses Unicode NFC normalisation, described as mapping variant codepoints to one form.^[2] We checked what NFC does to these letters, and it leaves Arabic Yeh and Farsi Yeh as two different characters. It also leaves ه followed by a separate hamza (U+0647 U+0654) different from the single character ۀ. NFC is needed, but it does not fix the Yeh problem on its own.

Folding can also go too far. In standard Pashto spelling, a final ی and a final ي are different letters with different sounds: سړی is “man” and سړي is “men”. A rule that treats every Yeh as the same character will score “men” as a correct transcript of “man”. And Whisper’s default text normaliser turns the separate hamza mark into a space, so پهٔ and په score as the same word. Both of these are our own checks of the Unicode and normaliser rules, not published results.

Metric labels need checking too. One public Pashto model card reported 13.96% WER. Rahman’s re-evaluation found 35.4% WER and 13.8% CER on the same test set, and concluded the card had reported character error rate under the name WER.^[2]

Before you compare Pashto scores, check whether they used NFC, which letters were folded together, and whether the number is WER or CER.

Published scores don’t hold on a new test set

Pashto ASR papers have mostly scored models on their own splits, and the results do not carry over.^[2] A Whisper-base model fine-tuned on Common Voice 20 reported 14% WER on its own test split.^{[2, 4]} On Common Voice 24, a later release of the same collection, it scored 35.0%. On FLEURS it scored 59.0%.^[2]

Models trained on FLEURS fell the other way. A fine-tuned Whisper-base went from 61.0% on FLEURS to 71.0% on Common Voice, and an XLS-R model from 51.9% to 70.7%.^[2] One W2V-BERT model trained with added noise and speed changes scored 35.1% on both sets, the only model with no drop.^[2]

Both test sets are read sentences. No one has measured how these models do on spontaneous Pashto.^[2] If your users speak freely, on the phone or in a clinic, expect results worse than any model card’s WER.

Dialects and who is in the data

Pashto’s dialects differ in exactly the sounds that cause errors. In the Kandahari standard, ښ and ږ are lateral fricatives. In Northern dialects they merge with other sibilants or stops.^[2] The dialects also differ in how they pronounce the retroflex series and vowels.^[2] So the ښ in our example sentence is written the same in Peshawar and Kandahar but does not sound the same.

We could not find WER broken down by Pashto dialect. The public sets do not allow it. Most Common Voice contributors leave the dialect field empty, so every Common Voice WER is a mix of dialects in unknown proportions.^[2] FLEURS gives no dialect information for any language.^{[1, 8]} The broadcast study used only Standard Afghan Pashto, and its authors list the single dialect as a limitation.^[3] The Pashto Common Voice builders think diaspora contributors probably over-represent Northern varieties.^[4]

The speakers in Common Voice are also narrow. An audit of release 24 found that 79% of clips come from speakers in their twenties and thirties, and speakers over sixty make up under 0.1%.^[5] Contribution is very uneven, with a Gini coefficient of 0.941 across contributors.^[5] 41.97% of clips have no gender label, and in a blind listening check of 300 unlabelled clips, many sounded male.^[5] For a model trained mainly on this data, an older speaker in Kandahar is far from the training set.

A telephone corpus with speech from four dialect regions of Afghanistan and Pakistan does exist: the IARPA Babel Pashto pack, recorded in 2011 and 2012 and sold through the Linguistic Data Consortium.^[7] We could not find a published test of current multilingual models on it.

Names, numbers and rare words

The Interspeech study measured these across Urdu, Punjabi, Pashto and Sindhi.^[1]

• Rare words: Pashto words seen at most twice in training were recognised 30 to 40 points less accurately than common words.^[1] Drug names and place names are rare words by this measure.
• Names: across the four languages, named entities scored 5 to 20 points below other words in most configurations.^[1]
• Numbers: on FLEURS, SeamlessM4T spelled out 87 to 89% of numerals as words, MMS wrote digits but got under a quarter right, and Whisper dropped 21 to 27% of them.^[1]
• English words: across the four languages, models dropped 27 to 62% of English words in the reference, transliterated 9 to 54% and kept only 1 to 11% in Latin script.^[1]
• Pashto-specific error profiles: Whisper-medium’s insertions reached 27% of its Pashto errors, nearly double its average, and MMS-1B’s deletions reached 19%, about twice its Urdu rate.^[1]

The study also notes that “Afghanistan” has several valid Pashto spellings, each of which WER counts as wrong against the others.^[1]

What helps

• Don’t use Whisper zero-shot for Pashto. Forcing the language did not make it write Pashto script.^[2] MMS, SeamlessM4T and OmniASR write Pashto script over 93% of the time.^[2]
• Check the script of every transcript. A check for Pashto-only letters such as ټ, ښ and ۍ catches wrong-script output that WER hides.^[2]
• Normalise before scoring, and choose the rules on purpose. Apply NFC, fold lookalike Kaf, Heh and Hamza forms, and check whether folding Yeh merges real Pashto words. Report raw and normalised WER together.^{[1, 2]}
• Test on two sets, not one. Scores from a model’s own split did not predict FLEURS or Common Voice 24.^[2]
• Fine-tune if you can. Fifteen hours of broadcast news took Whisper-small from 98.4% to 34.1%.^[3] Training with noise and speed changes removed the drop between test sets for one model.^[2]
• Write a policy for numbers and English words, and apply it to references and outputs alike.^[1]
• Confirm critical fields. For a dosage, an account number or an address, ask the speaker to confirm instead of trusting the transcript.

Open gaps

• Spontaneous and conversational Pashto results. Every public number is read speech, and we could not find a test of current models on the Babel telephone data.^{[2, 7]}
• WER by dialect. No public test set labels it.^[2]
• A Pashto clinical speech test set with consent, de-identification and clinician-checked references. We could not find one. The closest evidence is from other languages: on accented clinical English, models with low overall WER still made more errors on drug names and conditions.^[9]
• A shared Pashto normalisation standard that folds codepoint variants without merging distinct letters.
• Pashto mixed with Dari, Urdu or English. We could not find an ASR study of Pashto code-switching.
• Results from commercial speech APIs on a shared Pashto test set.

We plan to cover several of these in QuantClarity’s own Pashto evaluations.

References

1. A. H. Azeemi, I. A. Qazi, M. Mustafa, A. A. Raza. Dissecting ASR Failures in Low-Resource South Asian Languages. Interspeech 2026.
2. H. Rahman. Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation. arXiv preprint, 2026.
3. N. U. Sehar, A. Khalid, F. Adeeba, S. Hussain. Benchmarking Whisper for Low-Resource Speech Recognition: An N-Shot Evaluation on Pashto, Punjabi, and Urdu. CHiPSAL Workshop, COLING 2025.
4. H. Rahman, S. ur Rehman. Pashto Common Voice: Building the First Open Speech Corpus for a 60-Million-Speaker Low-Resource Language. Interspeech 2026.
5. J. Jahani, M. Dawodi, J. Baktash. From Scarcity to Scale: A Release-Level Analysis of the Pashto Common Voice Dataset. arXiv preprint, 2026.
6. H. Rahman. Fine-tuning Whisper for Pashto ASR: strategies and scale. arXiv preprint, 2026.
7. Appen. IARPA Babel Pashto Language Pack (LDC2016S09). Linguistic Data Consortium, 2016.
8. M. Lau et al.. Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning. ACL 2025.
9. T. Afonja, T. Olatunji, S. Ogun, N. A. Etori, A. Owodunni, M. Yekini. Performant ASR Models for Medical Entities in Accented Speech. Interspeech 2024.