Multilingual Utterances Transcription Benchmark — an open benchmark for evaluating speech-to-text providers across multiple locales and metrics.
Leaderboard · Dataset · Blog
This benchmark compares transcription providers on real customer service phone conversations recorded at 8kHz mono. Human annotators produce word-level ground truth transcripts for each caller utterance.
The dataset covers 5 locales with 4,270 utterances total:
| Locale | Language | Utterances |
|---|---|---|
| en-US | English (US) | 817 |
| es-MX | Spanish (Mexico) | 792 |
| tr-TR | Turkish | 846 |
| vi-VN | Vietnamese | 975 |
| zh-CN | Chinese (Mandarin) | 840 |
brew install ffmpeg # macOS
# or: apt-get install ffmpeg # LinuxThe audio is hosted as a gated dataset on HuggingFace. You'll need to:
- Request access to the
sierra-research/mu-benchdataset - Create a HuggingFace token and set it as an environment variable
export HF_TOKEN=your_token_here
pip install -e ".[tools]"
python scripts/download_audio.pyThis exports .wav files to audio/<locale>/.
cd leaderboard
npm install
npm run devmanifest.json contains every utterance in the benchmark with its ground truth transcript:
{
"id": "conv-0-turn-0",
"locale": "en-US",
"conversation_id": "conv-0",
"turn_index": 0,
"transcript": "Hi. I'm calling to check the status of my card.",
"audio_path": "audio/en-US/conv-0-turn-0.wav",
"duration_sec": 2.246
}See submissions/SUBMITTING.md for the full submission contract. The short version:
- Download the audio from the HuggingFace dataset
- Run your model on all audio files listed in
manifest.json - Save transcripts as
.txtfiles — one per utterance, plain text, no headers - Create
metadata.yamlwith your model info - Create
latency.jsonmapping"<locale>/<utterance_id>"to API response time in milliseconds - Validate locally with
python scoring/validate.py submissions/raw/your-model-name --manifest manifest.json - Open a pull request adding your submission under
submissions/raw/
See submissions/raw/deepgram-nova3/ for a complete example.
Scoring is run by maintainers — submitters do not need to run it locally. When a maintainer comments /score on your PR, the CI pipeline runs automatically:
| Metric | Description | Direction |
|---|---|---|
| WER | Corpus Word Error Rate (sum of word edits / sum of reference words) after LLM normalization, computed against a canonical per-utterance gold normalized once from manifest.json so every provider is scored against the same reference. Silent clips with non-<unintelligible> empty gold count any hypothesis words as insertion errors (both numerator and denominator receive the hyp word count). Overall is the unweighted mean of the five per-locale corpus WERs. |
Lower is better |
| UER | Utterance Error Rate — fraction of utterances with at least one meaning-changing error. Overall is the unweighted mean of the five per-locale UERs (same locale-macro convention as WER). | Lower is better |
| Latency (p95) | 95th percentile time-to-complete-transcript per utterance (ms). For batch APIs this is request-to-response round-trip; for streaming APIs it is send-to-final-transcript. Streaming submissions additionally surface TTFT as a +TTFT annotation. |
Lower is better |
The scoring pipeline is pinned for reproducibility: scoring/llm.py targets a specific judge model snapshot at temperature=0 with a fixed seed, and each scores.json records the model, seed, and prompt SHAs under a top-level judge block.
Results are posted as a comment on your PR and added to the leaderboard on merge.
manifest.json # Audio file list + ground truth transcripts
pyproject.toml # Python deps: base + [tools], [scoring], [transcribe] extras
scripts/
download_audio.py # Download audio from HuggingFace (requires [tools])
transcribe.py # Runs provider APIs to generate transcripts + latency (requires [transcribe])
latency_stats.py # Compute p50/p95 latency from latency.json files
submissions/
SUBMITTING.md # Full submission contract
raw/ # Raw transcripts + latency.json, one directory per provider
normalized/ # LLM-normalized transcripts (auto-generated by CI)
scoring/
validate.py # Submission format validation (run locally)
normalize.py # LLM-based transcript normalization
score.py # Metrics computation (corpus WER, sig. WER)
metrics.py # Core metric implementations
prompts.py # Gitignored — injected from GitHub secret in CI
results/
leaderboard.json # Aggregated leaderboard data
<provider>/scores.json # Per-provider score breakdowns (includes latency)
leaderboard/ # React/Vite leaderboard web app
.github/workflows/ # CI for validation, scoring, and deployment
The project uses pyproject.toml with three optional extras:
pip install -e . # base: just the validator (pyyaml)
pip install -e ".[tools]" # + huggingface-hub, requests (for download_audio.py)
pip install -e ".[scoring]" # + jiwer, openai (maintainers; needs scoring/prompts.py)
pip install -e ".[transcribe]" # + provider SDKs (for running scripts/transcribe.py)If you use MU-Bench in your research, please cite:
@misc{mubench2026,
title = {MU-Bench: A Multilingual Transcription Benchmark from Real Phone Calls},
author = {Li, Andrea and Ray, Soham},
year = {2026},
url = {https://github.com/sierra-research/mu-bench},
note = {Dataset: https://huggingface.co/datasets/sierra-research/mu-bench}
}This repository uses a dual-license model:
- Code (scripts, scoring, leaderboard, workflows, and all other software) is licensed under the Apache License 2.0.
- Data (audio files and transcripts from Hugging Face and manifest.json) is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).