Analyzing Open AI's Whisper ASR Accuracy: Word Error Rates Across Languages and Model Sizes

While OpenAI has published Whisper accuracy numbers for some English open source data sets, there is relatively little information on performance for other languages. Furthermore, the most common open source benchmarks, such as Common Voice and LibriSpeech, are rather clean audio, captured in relatively good acoustic conditions, and contain well articulated speech. Transcription in real life use cases is typically messier. The audio often has poor acoustic conditions and articulation, thick accents, hesitation, overlapping speech, and so on.  These factors all made it attractive to conduct a more robust analysis of Whisper performance across model sizes, languages, and audio quality.

To test the models, we manually transcribed 5 hours' worth of YouTube videos in different languages to establish the ground truth. Youtube videos naturally contain the aforementioned “messiness” and therefore, the word error rates (WER) obtained with Youtube are perhaps a better proxy, compared to an open source benchmark, to what you might expect from typical in-the-wild transcription scenarios. We used the youtube data to test different-sized Whisper multilingual speech recognition models, comparing their transcripts to the ground truths to calculate WER. We also computed the relative word error rate reduction between Whisper small and medium, denoted WERR: S → M.

The resulting word error rates are presented in the table below:

* Character error rate in stead of word error rate.

The top-performing languages for Whisper transcription accuracy are English, Italian, German, and Spanish. Mid-performing languages include French, Portuguese, and Japanese, while the worst-performing languages are Arabic and Hindi.

It is worth noting that the small model often offers the best value for money. There are only slight gains in running the large or medium models in most languages. However, there are some exceptions where the medium model does provide relevant accuracy gains. Languages such as Italian, Danish, Greek, Thai, Tagalog, and Finnish show a noticeable improvement in accuracy when using the medium model compared to the small model.

Additionally, the large model does not provide significant accuracy gains over the medium or small models for most languages. This suggests that, in general, the small and medium models offer the  balance between cost and performance.

* Actually, Whisper does offer Dutch, but we just couldn't resist the temptation 😎