The usual sequence goes like this. Whisper looks fine in English, you point it at a Norwegian, Slovak or Greek workload, the word error rate roughly doubles, and someone opens a fine-tuning notebook.
That is the wrong second step. Fine-tuning is one intervention among five or six, it is the most expensive one, and it only helps against some of the failures you are seeing. Teams that skip the measurement step usually spend a quarter discovering which of their errors fine-tuning was never going to touch.
This post is the workflow that comes first: profile the errors, rank the interventions by cost, and refuse to ship any of them without a regression gate.
Three failures that look identical in an aggregate WER
A single WER number over a mixed evaluation set hides the only thing you need to know, which is which kind of error you have. At minimum, separate these three before choosing a remedy.
Hallucinations. Invented segments appear around silence, background noise, and weak or distant speech. They are not a language problem at all, they are a decoding and voice-activity problem, and they respond to preprocessing and decoding constraints far more cheaply than to retraining. They also inflate WER in a way that makes a dialect problem look worse than it is.
Dialect and domain shift. Regional variation, in-domain vocabulary and speaker conditions move error patterns independently of each other. Norwegian is the sharpest illustration in the published evaluations: performance on standard read speech tells you very little about performance on regional spontaneous speech. The numbers for that gap are in Whisper Hits 30% WER on Spoken Norwegian in Every Dialect, and the cross-language picture is in the European Dialect ASR Benchmark.
Output-contract failures. Punctuation, casing, paragraphing, normalization, terminology and timestamp quality are frequently the errors that actually break the downstream product, and they are invisible in WER because most normalization pipelines strip exactly those features before scoring. If your consumer is subtitles, search or redaction, a transcript can be WER-excellent and still unusable.
Lumping these together produces the most common wasted cycle in ASR work: a fine-tuning run that improves substitution rates by a few points while the hallucination and formatting failures that were breaking production go untouched.
Build a workload error profile
The evaluation set is the entire foundation, and a public benchmark is not one. It has to be representative of the audio you actually process.
Break the results down along every axis that varies in your workload:
- Language and regional variant. Not “Norwegian” but which varieties, in what proportion.
- Acoustic condition. Recording chain, noise floor, distance, channel, codec.
- Domain. The vocabulary and register the model has to survive.
- Speaker. Overlap, turn-taking, attribution requirements, demographics.
- Failure type. Hallucination, substitution, omission, formatting, diarization, timestamp.
The output is a table, not a number. That table tells you which intervention is worth buying, and it is the only artifact that will later let you prove a change helped.
Two practical constraints. Reference transcripts have to follow one written convention, or you will be measuring your annotators’ disagreement rather than the model. And the set has to be versioned, because you will compare against it for the life of the system.
Rank the interventions by cost, not by novelty
Once the profile exists, the options fall into a rough cost order. Work down it, not up.
- Decoding settings and voice-activity handling. Temperature, fallback behaviour, no-speech thresholds, condition-on-previous-text, segment boundaries. This is the cheapest lever and it is the one that moves hallucination rates.
- Preprocessing. Resampling, channel handling, noise handling, segmentation. Cheap, reversible, and frequently the real fix when the acoustic condition is the variable.
- Vocabulary and prompting. Domain terminology, initial prompts, biasing where your runtime supports it.
- Model adaptation. Fine-tuning or a different base model. Genuinely powerful against dialect and domain shift, and genuinely expensive. It needs data, and if your target variety is thin in the base model, it may need collected data rather than scraped data. That decision is its own subject, covered in Beyond Whisper: Custom Speech Data for Low-Resource ASR.
- Human review. Confidence-routed correction for the residue. Not a failure of engineering; for high-stakes output it is usually the only honest answer for the last few percent.
Most European-language workloads we see have meaningful gains available in the first three before anyone touches a training run.
Gate every change on a regression test
The step teams skip. Before a change reaches the production workflow, three things have to exist:
- Acceptance thresholds, defined per failure type and per condition rather than as one aggregate number.
- A repeatable test, run from a versioned evaluation set with recorded normalization and decoding settings.
- A record of what was tried and rejected, so the next engineer does not repeat the same experiment.
Without this you cannot distinguish a real improvement from evaluation-set drift, and you cannot tell whether a fix for one dialect quietly regressed another. Anything less means each change is a bet rather than a measurement.
Reproducibility is the boring half of this and the half that matters: dataset version, split, references, normalization rules, decoding settings, metrics, and the review decisions behind them. If it is not recorded, the result is not a result.
Streaming, timestamps, and the things WER does not see
If you run streaming rather than batch, the evaluation has to match: latency, throughput, stability under concurrency, and quality using the real codecs, packetization and network path. Batch WER predicts streaming behaviour poorly.
If timestamps feed subtitles, search or redaction, measure timestamp quality directly against that use case. And put punctuation, casing, paragraphing, normalization and terminology into the acceptance contract explicitly, because the scoring pipeline will otherwise discard exactly the properties your product depends on.
Settle the data boundary before audio moves
Speech is personal data, and in most European deployments this is the constraint that decides the architecture rather than an afterthought to it. Four questions need documented answers before any audio is handled:
- Where processing and storage happen.
- Who has access, and what transfers occur.
- How long data is retained.
- Whether the audio may be used for model adaptation at all.
The last one matters most here, because “improve the model” and “use your recordings to do it” are separate permissions, and they need separate answers in writing. Controls should map to the architecture that actually exists, not to a generic policy.
The short version
Measure before you adapt. Separate hallucination from dialect shift from output-contract failures, because they have different and differently-priced remedies. Work down the cost order rather than starting at fine-tuning. Gate everything on a versioned regression test. And document the data boundary before the first file moves.
If you want the measured picture for Nordic and wider European varieties, start with the European Dialect ASR Benchmark. If you are scoping an evaluation of your own workload, the speech evaluation program describes how we set one up.
Sources: