--- title: European Dialect ASR Benchmark (Q3 2026) url: https://ypai.ai/blog/data-engineering/european-dialect-asr-benchmark/ category: Data Engineering published: 2026-07-24T00:00:00.000Z modified: 2026-07-24T00:00:00.000Z author: YPAI Engineering tags: [ASR, Whisper, Dialects, Benchmarks, Speech Data] --- # European Dialect ASR Benchmark (Q3 2026) > Every published dialect WER result for European languages 2023-2026: Norwegian, Danish, Swedish, Swiss German. Primary sources only, updated quarterly. Standard ASR benchmarks measure how well a model transcribes the written norm of a language. They say almost nothing about the Nynorsk speaker, the Walser dialect, or a Jutland accent, and that silence is where production speech systems fail. This page collects every dialect-specific European ASR result we could verify against a primary source, published 2023-2026, in one citable place. Three rules govern this benchmark. Every number links to its primary source: a paper, peer-reviewed proceedings, or the publishing lab's official model card. Results reported elsewhere that we could not verify at the primary source are listed separately, not mixed in. And the two evaluation paradigms in this field are never compared against each other, because their WER numbers measure different things. ## Two paradigms, two tables **Dialect transcription** evaluates audio against a reference in an orthography of the same variety. Norwegian, Danish, and Swedish results work this way. **Dialect-to-standard translation** evaluates dialect audio against Standard German text, because Swiss German has no standardized orthography. WER here partly measures translation convention, not recognition, which is why the field also reports BLEU and content-WER (cWER) for these systems. Comparing a Swiss German 25% against a Danish 28% is a category error. ## Verified results: dialect transcription | Language | Variety | Model | Test set | WER | Source | |---|---|---|---|---|---| | Norwegian | Bokmaal, read speech | OpenAI Whisper large-v3 | NST | 6.8% | [Whispering in Norwegian (arXiv 2402.01917)](https://arxiv.org/abs/2402.01917) | | Norwegian | Bokmaal, read speech | NB-Whisper Large | NST | 2.2% | [arXiv 2402.01917](https://arxiv.org/abs/2402.01917) | | Norwegian | Bokmaal | OpenAI Whisper large-v3 | FLEURS | 10.4% | [arXiv 2402.01917](https://arxiv.org/abs/2402.01917) | | Norwegian | Bokmaal | NB-Whisper Large | FLEURS | 6.6% | [arXiv 2402.01917](https://arxiv.org/abs/2402.01917) | | Norwegian | Nynorsk | OpenAI Whisper Large | Common Voice | 30.0% | [arXiv 2402.01917, Table 7](https://arxiv.org/html/2402.01917v1) | | Norwegian | Nynorsk | NB-Whisper Large | Common Voice | 12.6% | [arXiv 2402.01917, Table 7](https://arxiv.org/html/2402.01917v1) | | Danish | All major dialects | OpenAI Whisper large-v3 | CoRal test | 28.3% | [Roest-315m model card](https://huggingface.co/alexandrainst/roest-315m) | | Danish | All major dialects | OpenAI Whisper large-v2 | CoRal test | 32.6% | [Roest-315m model card](https://huggingface.co/alexandrainst/roest-315m) | | Danish | All major dialects | OpenAI Whisper medium | CoRal test | 40.5% | [Roest-315m model card](https://huggingface.co/alexandrainst/roest-315m) | | Danish | All major dialects | OpenAI Whisper small | CoRal test | 55.2% | [Roest-315m model card](https://huggingface.co/alexandrainst/roest-315m) | | Danish | All major dialects | Roest wav2vec2-315m | CoRal test | 17.0% | [Roest-315m model card](https://huggingface.co/alexandrainst/roest-315m) | | Swedish | Broad corpus incl. NST dialect data | KB-Whisper (best) vs large-v3 | FLEURS + CV + NST | 47% avg. relative WER reduction | [Swedish Whispers (arXiv 2505.17538)](https://arxiv.org/abs/2505.17538) | Details worth noting from the primary sources: on Nynorsk, OpenAI Whisper models below medium exceed 100% WER in the paper's Table 7, and the Danish CoRal numbers are bootstrapped means with 95% confidence intervals published on the model card. ## Verified results: Swiss German (dialect-to-standard translation) | Variety | Model | Test set | WER | Source | |---|---|---|---|---| | All dialects | Whisper large-v3, zero-shot | STT4SG-350 | 23.0% | [Dolev et al., VarDial 2024](https://aclanthology.org/2024.vardial-1.3.pdf) | | All dialects | XLS-R, fine-tuned (Schraner 2022) | STT4SG-350 | 15.3% | [compiled in VarDial 2024, Table 2](https://aclanthology.org/2024.vardial-1.3.pdf) | | All dialects | wav2vec2, fine-tuned (Pluess 2023) | STT4SG-350 | 14.0% | [compiled in VarDial 2024, Table 2](https://aclanthology.org/2024.vardial-1.3.pdf) | | Per dialect, best: Grisons | Whisper large-v3, zero-shot | SwissDial | 16.9% | [VarDial 2024, Table 4](https://aclanthology.org/2024.vardial-1.3.pdf) | | Per dialect, worst: Walser | Whisper large-v3, zero-shot | SwissDial | 29.7% | [VarDial 2024, Table 4](https://aclanthology.org/2024.vardial-1.3.pdf) | | All dialects | Whisper large-v3, full fine-tune | ASGDTS (disjoint) | 25.60% (13.8% cWER) | [arXiv 2606.07608](https://arxiv.org/abs/2606.07608) | | All dialects | Whisper large-v3, LoRA | ASGDTS (disjoint) | 25.32% (13.9% cWER) | [arXiv 2606.07608](https://arxiv.org/abs/2606.07608) | The SwissDial per-dialect spread is the single most instructive row pair in this collection: the same model on the same language ranges from 16.9% to 29.7% WER depending only on which dialect is speaking. Any acceptance test that reports one aggregate number hides that spread. ## Three lessons the primary sources agree on **1. National fine-tunes beat vanilla Whisper everywhere they exist.** Norwegian NB-Whisper cuts NST WER from 6.8% to 2.2% and Nynorsk from 30.0% to 12.6%. Danish Roest cuts CoRal WER from 28.3% to 17.0%. Sweden's KB-Whisper reports an average 47% relative reduction. The pattern is uniform: targeted, dialect-diverse training data outperforms model scale, the same conclusion we reached in our [Whisper dialect analysis for Scandinavian languages](/blog/data-engineering/whisper-fails-scandinavian-dialects-asr-benchmark/). **2. Model size does not rescue dialects.** On Danish CoRal, WER degrades monotonically as Whisper shrinks: 28.3% (large-v3), 32.6% (large-v2), 40.5% (medium), 55.2% (small). On Nynorsk, sub-medium models are unusable. If your deployment budget forces a smaller model, dialect-specific fine-tuning data stops being optional. **3. Trust disjoint test sets only.** The 2026 Swiss German study documented that a vanilla Whisper model self-trained on the test set, with zero Swiss German data, beat all previously published systems at 13.88% WER, showing that the old benchmark rewarded convention matching, not comprehension. The honest baseline on strictly disjoint data is 25.6% WER. When a vendor quotes a dialect WER, the first question is whether the test set was disjoint from training. ## The gap map: where no published numbers exist For the following priority varieties we found no published, dialect-specific ASR evaluation as of Q3 2026. Standard-norm test sets (FLEURS, Common Voice) exist for the languages, but the regional performance is unmeasured in public literature: - Norwegian spoken regional dialects (as opposed to the two written norms) - Austrian German and German regional dialects other than Swiss German - Belgian and Swiss French - Andalusian, Canarian, and Murcian Spanish - Italian regional languages: Neapolitan, Sicilian, Lombard, Venetian - Cypriot, Cretan, and Pontic Greek - Regional European Portuguese: Azorean, Madeiran - Silesian and Goral Polish - Moldavian and Transylvanian Romanian - Colloquial spoken Finnish (puhekieli) - Icelandic regional accents - Dutch regional varieties: West Flemish, Limburgish Every entry on this list is a publishable research contribution waiting to happen: an open, disjoint, dialect-specific test set with baseline numbers would be the first citable reference for that variety. ## Reported but not yet verified These results circulate in reports and secondary sources but have not yet passed our primary-source verification; they will be added if they do: Flemish conversational vs broadcast Whisper results on N-Best 2008 (source PDF currently inaccessible), the Spaiche Whisper-medium SDS-200 result, the Roest-Whisper-large Danish result, German Betthupferl regional results, and Irish BlasBench. Nothing in this section should be cited from this page. ## Using this in procurement If your users speak a dialect, your acceptance criteria should name it. The published spread within a single language (16.9% to 29.7% for Swiss German; 6.8% to 30.0% between Norwegian norms) means an aggregate WER commitment is not a dialect commitment. Require per-variety evaluation on a disjoint test set, and if no public test set exists for your variety (see the gap map), a [custom dialect corpus](/blog/data-engineering/beyond-whisper-custom-speech-data-low-resource-languages/) is the prerequisite, not a nice-to-have. Our [language and dialect coverage](/speech-data/language-coverage/) page describes how we scope that per engagement. This page is updated quarterly as new primary-source results are published. Last verified: 2026-07-24. --- ## Related Resources - [Whisper fails outside standard Norwegian: the real numbers](/blog/data-engineering/whisper-fails-scandinavian-dialects-asr-benchmark/) - Our analysis of the Scandinavian dialect gap and what fixes it - [Beyond Whisper: custom speech data for low-resource ASR](/blog/data-engineering/beyond-whisper-custom-speech-data-low-resource-languages/) - When fine-tuning data beats model scale - [Improve Whisper for European languages](/solutions/fixing-whisper-european-languages/) - Engineering approach to dialect adaptation - [ASR software comparison](/blog/data-engineering/asr-software-comparison/) - How the major engines differ on European languages - [Language and dialect coverage](/speech-data/language-coverage/) - Per-engagement dialect scoping with controlled validation - [Speaker diarization training data requirements](/blog/data-engineering/speaker-diarization-training-data-requirements/) - Corpus design for multi-speaker dialect data