Key Takeaways
- What speech recognition software delivers in controlled lab conditions and what it delivers in production are routinely different. Dialect coverage, background noise, and domain vocabulary are the three most common gaps.
- Cloud ASR APIs (Google, Azure, AWS) reduce infrastructure burden but tie accuracy and cost to vendor roadmaps. Open-source models like Whisper give control but require significant investment in fine-tuning and hosting.
- Word error rate alone is not a useful evaluation metric. WER on benchmark audio systematically overpredicts accuracy on real production speech.
- The training data used to build or fine-tune an ASR model determines its accuracy ceiling. No post-processing step recovers accuracy that the training corpus never provided.
- Multilingual and dialect coverage requirements eliminate most off-the-shelf options for European enterprise deployments. Custom corpora are often the only path to production-grade performance.
What speech recognition software actually does in production is rarely what benchmarks suggest. Enterprise teams evaluating ASR engines encounter a common pattern: strong published accuracy numbers, credible vendor demonstrations, and then a materially different experience once real users with real accents, real background noise, and real domain vocabulary start talking.
The gap is not always a vendor honesty problem. It is a benchmark problem. Standard ASR benchmarks measure clean, read speech from a narrow demographic. Production speech is none of those things.
This article covers what speech recognition engine categories exist, what the evaluation criteria actually measure versus what they predict, and where the training data problem determines the accuracy ceiling before any other factor.
What speech recognition software does
ASR software converts audio input into text. The conversion happens through an acoustic model that maps audio features to phonemes, a language model that assigns probability to word sequences, and a decoder that finds the most likely transcription. Modern end-to-end neural architectures combine these stages into a single model, but the underlying problem is unchanged: recognising what was said from a continuous audio signal.
The difficulty varies by acoustic conditions, speaker characteristics, and vocabulary domain. Quiet, single-speaker recordings of standard English follow predictable statistical patterns that large training sets cover well. Multi-speaker, accented, domain-specific audio in a noisy environment does not. The distribution shift between training conditions and deployment conditions is the primary source of production ASR failures.
The main engine categories
Enterprise ASR deployment options divide into three categories, each with a different set of tradeoffs.
Cloud ASR APIs
Google Cloud Speech-to-Text, Microsoft Azure AI Speech, AWS Transcribe, and Deepgram represent the commercial cloud API tier. The operational model: send audio to an API endpoint, receive text in return. Infrastructure, model training, and updates are the vendor’s problem. The tradeoffs are data residency, cost at scale, latency, and the accuracy boundaries the vendor’s training data imposes.
Cloud APIs perform well for the languages and domains their training corpora cover densely. Major European languages spoken by speakers with standard accents in low-noise conditions typically fall within this category. Regional dialects, accented speech from non-native speakers, and domain-specific vocabulary in less-resourced languages frequently do not.
Vendor pricing varies significantly by usage volume and feature tier. Real-time streaming APIs carry different pricing from batch transcription. Speaker diarization, word-level timestamps, and domain adaptation (custom vocabulary or model fine-tuning) are typically priced separately from base transcription.
Open-source models
OpenAI Whisper is the dominant open-source option following its 2022 release and subsequent large-v3 update. Trained on 680,000 hours of web-collected multilingual audio, Whisper covers a wider language range than most commercial APIs. The model weights are public, which allows fine-tuning on domain-specific corpora without sending audio to a vendor. The operational model: download the model, run inference on your own infrastructure.
The tradeoffs are infrastructure cost and latency. Whisper large-v3 requires a capable GPU for real-time or near-real-time transcription. Batch processing is feasible on more modest hardware, but with processing times that exclude real-time applications. Hosting, serving, and maintaining the model is an engineering cost that cloud APIs absorb.
Meta’s MMS (Massively Multilingual Speech) and NVIDIA NeMo provide additional open-source options with different architectural choices and training data provenance. For multilingual deployments, model architecture choice interacts with available fine-tuning data in ways that make single-engine recommendations unreliable.
Self-hosted commercial engines
Assembly AI, Rev AI, and Speechmatics sit between cloud APIs and open-source models. They offer more deployment flexibility than standard cloud APIs, including on-premise options that address data residency requirements, while reducing the infrastructure burden of self-hosted open-source deployment. This tier is most relevant when privacy requirements rule out standard cloud APIs but GPU infrastructure investment is not viable.
Key evaluation criteria
Accuracy on your data, not benchmark data
Word error rate is the standard accuracy metric, calculated as the number of incorrect words divided by the total reference words. Published WER scores on standard benchmarks (LibriSpeech, Common Voice, Fleurs) provide a relative ranking of models on well-defined test conditions. They do not predict accuracy on your deployment speech.
The evaluation that matters is WER measured on held-out samples from your actual user population, in your target acoustic conditions, using your target domain vocabulary. Request this evaluation from vendors. Provide your own audio samples. Treat any vendor that will not perform this evaluation as a risk.
Latency and streaming support
Real-time transcription applications require streaming ASR with low latency. Batch transcription of recorded audio tolerates higher latency. The latency requirements determine which models are viable: large Whisper variants are not practical for real-time streaming without substantial GPU investment. Cloud APIs vary by tier in their latency guarantees.
Latency measurements must be taken end-to-end from audio input to usable text output, including network round-trips for cloud APIs. In-region deployment reduces latency but may constrain model choice.
Multilingual and dialect coverage
What speech recognition software delivers for major European languages with standard accents is not the same as what it delivers for regional dialects, code-switched speech, or accented non-native speakers of those languages. The distinction matters for European enterprise deployments where speaker populations are not linguistically homogeneous.
Whisper’s broad multilingual training gives it an advantage in language coverage, but accuracy for specific dialects and accented speech still requires evaluation. Commercial APIs typically focus training investment on high-volume languages and language varieties. For deep Nordic coverage, Iberian regional varieties, or Eastern European languages outside the major tier, evaluate specifically before committing.
Cost at scale
Cloud API pricing for transcription scales with audio minutes processed. At low volume, managed APIs are cost-efficient. At high volume, the comparison with self-hosted open-source models shifts: GPU infrastructure is a fixed cost, while API costs scale linearly. The break-even point depends on volume, model size requirements, and infrastructure costs in the deployment region.
Privacy and data residency
Audio sent to a cloud API is processed on the vendor’s infrastructure. For European deployments under GDPR, processing personal voice data outside the EEA requires Standard Contractual Clauses and Transfer Impact Assessments. Regulated industries, healthcare applications, and applications processing sensitive content may have requirements that standard cloud API terms do not satisfy. Self-hosted deployment, whether open-source or commercial on-premise, keeps audio within your infrastructure.
Where ASR fails and why
The failure patterns of production ASR systems are consistent regardless of engine choice.
Dialect and accent gaps. Models trained on data that does not represent the target speaker population underperform on those speakers. A Norwegian Bokmål model trained primarily on Oslo speech will fail on Nynorsk and regional dialects. This is not a model limitation that better architecture resolves. It is a training data gap that only representative training data resolves.
Background noise and recording conditions. Clean close-microphone speech is overrepresented in most training corpora. Speech captured by laptop microphones in office environments, mobile phones in transit, or call centre headsets introduces noise profiles the model has not learned. Acoustic model robustness requires training data that includes the target recording conditions.
Domain-specific vocabulary. Medical terminology, legal language, technical jargon, and product names appear rarely in general web-collected audio. Low-frequency vocabulary produces high substitution errors regardless of acoustic quality. Domain adaptation via fine-tuning or custom vocabulary lists addresses this, but requires representative domain audio.
Multi-speaker and overlapping speech. Speaker diarization (identifying who spoke which segment) is a separate task from transcription. Most ASR models are trained on single-speaker audio. Overlapping speech and rapid speaker changes degrade both transcription and diarization accuracy.
The role of training data in ASR accuracy
Training data determines the accuracy ceiling of any ASR engine. No post-processing step, language model overlay, or confidence scoring recovers accuracy that the acoustic model never learned. This is the most consequential fact for enterprise ASR deployment.
For off-the-shelf models and APIs, the training data is fixed. The vendor’s training corpus determines which language varieties, acoustic conditions, and vocabulary domains the model handles accurately. Fine-tuning on domain-specific data adjusts the model’s distribution, but the quality and representativeness of the fine-tuning corpus determines how much improvement is achievable.
For teams building custom models or fine-tuning open-source models on domain-specific data, the corpus specification is the primary engineering decision. More audio hours help, but representative coverage matters more than volume. A fine-tuning corpus that accurately represents target speaker demographics, acoustic conditions, and domain vocabulary will outperform a larger corpus that does not.
Representative training data for European enterprise ASR requires: speakers from the target linguistic regions with documented dialect coverage; balanced demographics across age, gender, and language background; acoustic conditions that match deployment environments; and domain-specific vocabulary coverage at sufficient frequency for the model to learn reliable pronunciations and sequences.
This is why YPAI collects speech data across European languages using a network of verified contributors in the EEA. Human-verified corpora with 50+ EU dialect coverage and documented consent address the training data gaps that off-the-shelf models leave.
For the engineering decisions upstream of ASR engine selection, see our guide to AI training data requirements and the detailed treatment of corpus design in our speech corpus collection for enterprise ASR guide.
Choosing based on your requirements
The engine selection decision simplifies when requirements are stated precisely.
For standard languages, moderate volume, and low-friction deployment: cloud APIs cover the requirement. Evaluate on your specific audio before committing, but the infrastructure advantage is real for teams without ML engineering capacity.
For privacy-constrained deployments, non-standard languages, or dialect-heavy user populations: open-source fine-tuning is typically the path. The infrastructure investment is unavoidable, but the accuracy achievable on representative training data exceeds what cloud APIs deliver for difficult language varieties.
For regulated industries where both privacy and managed reliability matter: commercial self-hosted or private cloud options bridge the gap, at a cost premium.
What all three categories share: accuracy on production speech is determined by training data coverage. The engine architecture matters less than whether the model has seen speech that resembles what your users produce. The audio annotation pipeline for speech data labeling determines the quality of any corpus used for fine-tuning, which directly determines what accuracy the fine-tuned model achieves.
Getting started
The right ASR engine evaluation starts with your actual speech samples, not vendor benchmarks. Collect 20-50 representative recordings from your target user population under your target acoustic conditions. Use those samples to benchmark every engine under consideration. The results will differ from published benchmarks, and that difference is the information that matters.
If the evaluation reveals accuracy gaps driven by dialect coverage, domain vocabulary, or speaker demographics that off-the-shelf models do not address, the path forward is fine-tuning on a representative corpus.
YPAI works with enterprise data teams to specify and collect fine-tuning corpora that match deployment requirements. EEA-only collection, 50+ dialect coverage, human-verified transcriptions, and EU AI Act Article 10 documentation are standard across our speech data services. If you are evaluating ASR engines and finding accuracy gaps that training data could resolve, contact our data team to discuss corpus requirements.
Sources:
