Skip to content
sierra-researchPublic

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

20 stars

Watchers

1 watching

Forks

Repository files navigation

MU-Bench

License: Apache 2.0 License: CC BY-NC 4.0 CI Dataset on HF

Multilingual Utterances Transcription Benchmark — an open benchmark for evaluating speech-to-text providers across multiple locales and metrics.

Leaderboard · Dataset · Blog

Overview

This benchmark compares transcription providers on real customer service phone conversations recorded at 8kHz mono. Human annotators produce word-level ground truth transcripts for each caller utterance.

The dataset covers 5 locales with 4,270 utterances total:

Locale Language Utterances
en-US English (US) 817
es-MX Spanish (Mexico) 792
tr-TR Turkish 846
vi-VN Vietnamese 975
zh-CN Chinese (Mandarin) 840

Getting Started

1. Install System Dependencies

brew install ffmpeg   # macOS
# or: apt-get install ffmpeg   # Linux

2. Download Audio

The audio is hosted as a gated dataset on HuggingFace. You'll need to:

  1. Request access to the sierra-research/mu-bench dataset
  2. Create a HuggingFace token and set it as an environment variable
export HF_TOKEN=your_token_here
pip install -e ".[tools]"
python scripts/download_audio.py

This exports .wav files to audio/<locale>/.

3. View the Leaderboard Locally

cd leaderboard
npm install
npm run dev

4. Explore the Manifest

manifest.json contains every utterance in the benchmark with its ground truth transcript:

{
  "id": "conv-0-turn-0",
  "locale": "en-US",
  "conversation_id": "conv-0",
  "turn_index": 0,
  "transcript": "Hi. I'm calling to check the status of my card.",
  "audio_path": "audio/en-US/conv-0-turn-0.wav",
  "duration_sec": 2.246
}

Submitting Results

See submissions/SUBMITTING.md for the full submission contract. The short version:

  1. Download the audio from the HuggingFace dataset
  2. Run your model on all audio files listed in manifest.json
  3. Save transcripts as .txt files — one per utterance, plain text, no headers
  4. Create metadata.yaml with your model info
  5. Create latency.json mapping "<locale>/<utterance_id>" to API response time in milliseconds
  6. Validate locally with python scoring/validate.py submissions/raw/your-model-name --manifest manifest.json
  7. Open a pull request adding your submission under submissions/raw/

See submissions/raw/deepgram-nova3/ for a complete example.

Scoring

Scoring is run by maintainers — submitters do not need to run it locally. When a maintainer comments /score on your PR, the CI pipeline runs automatically:

Metric Description Direction
WER Corpus Word Error Rate (sum of word edits / sum of reference words) after LLM normalization, computed against a canonical per-utterance gold normalized once from manifest.json so every provider is scored against the same reference. Silent clips with non-<unintelligible> empty gold count any hypothesis words as insertion errors (both numerator and denominator receive the hyp word count). Overall is the unweighted mean of the five per-locale corpus WERs. Lower is better
UER Utterance Error Rate — fraction of utterances with at least one meaning-changing error. Overall is the unweighted mean of the five per-locale UERs (same locale-macro convention as WER). Lower is better
Latency (p95) 95th percentile time-to-complete-transcript per utterance (ms). For batch APIs this is request-to-response round-trip; for streaming APIs it is send-to-final-transcript. Streaming submissions additionally surface TTFT as a +TTFT annotation. Lower is better

The scoring pipeline is pinned for reproducibility: scoring/llm.py targets a specific judge model snapshot at temperature=0 with a fixed seed, and each scores.json records the model, seed, and prompt SHAs under a top-level judge block.

Results are posted as a comment on your PR and added to the leaderboard on merge.

Repository Structure

manifest.json              # Audio file list + ground truth transcripts
pyproject.toml             # Python deps: base + [tools], [scoring], [transcribe] extras
scripts/
  download_audio.py        # Download audio from HuggingFace (requires [tools])
  transcribe.py            # Runs provider APIs to generate transcripts + latency (requires [transcribe])
  latency_stats.py         # Compute p50/p95 latency from latency.json files
submissions/
  SUBMITTING.md            # Full submission contract
  raw/                     # Raw transcripts + latency.json, one directory per provider
  normalized/              # LLM-normalized transcripts (auto-generated by CI)
scoring/
  validate.py              # Submission format validation (run locally)
  normalize.py             # LLM-based transcript normalization
  score.py                 # Metrics computation (corpus WER, sig. WER)
  metrics.py               # Core metric implementations
  prompts.py               # Gitignored — injected from GitHub secret in CI
results/
  leaderboard.json         # Aggregated leaderboard data
  <provider>/scores.json   # Per-provider score breakdowns (includes latency)
leaderboard/               # React/Vite leaderboard web app
.github/workflows/         # CI for validation, scoring, and deployment

Installing dependencies

The project uses pyproject.toml with three optional extras:

pip install -e .                # base: just the validator (pyyaml)
pip install -e ".[tools]"       # + huggingface-hub, requests (for download_audio.py)
pip install -e ".[scoring]"     # + jiwer, openai (maintainers; needs scoring/prompts.py)
pip install -e ".[transcribe]"  # + provider SDKs (for running scripts/transcribe.py)

Citing MU-Bench

If you use MU-Bench in your research, please cite:

@misc{mubench2026,
  title     = {MU-Bench: A Multilingual Transcription Benchmark from Real Phone Calls},
  author    = {Li, Andrea and Ray, Soham},
  year      = {2026},
  url       = {https://github.com/sierra-research/mu-bench},
  note      = {Dataset: https://huggingface.co/datasets/sierra-research/mu-bench}
}

License

This repository uses a dual-license model:

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

20 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages