📖 Quranic ASR Leaderboard
A leakage-free, held-out benchmark of Arabic ASR systems on Quranic recitation.
⭐ Head-to-head vs Tarteel's official realtime model: all 600 clips through Tarteel's production ASR (voice-v2.tarteel.io), same scorer: 9.56 overall WER (benchmark v1.1). The overall leader is zipformer_p-arabic-v3, a streaming phoneme model, at 3.46* WER; the best open-vocabulary system is our offline FastConformer at 4.13. Streaming field: v3 (3.46*), v2 (5.64*), zipformer_p-quran (5.83*), Tarteel (9.56), streaming FastConformer (10.53). *Closed-vocab nearest-ayah retrieval — the PER column and the "(PER)" subset cells give the phoneme models' native accuracy; v2/p-quran WER still scored against v1.0 references, re-run pending. 🤗 Benchmark dataset · 🚀 Submit your model
Every model is evaluated on the same 600 held-out clips (200 per source) with the same scorer; every clip is verified absent from our training data. Rankings use overall WER on all 600 clips (lower is better).
Rank | Model | Mode | Size (B) | Overall | PER | Phone (real-world) | EveryAyah | QUL (unseen reciter) | WER alef-ins. |
|---|---|---|---|---|---|---|---|---|---|
🥇 1 | 🟢 Streaming | 0.0655 | 10.32 | 11.54 | 22.75 | 12.39 | 11.91 | 10.14 |
Overall spans all 600 clips; per-source columns are 200 clips each. Lower is better.
Test sources
📱 Phone / real-world (tlog) — 200 clips
Held-out real phone-mic recitations. The hard case, and the one that matters for apps — studio numbers flatter every model; this column does not.
🎙️ EveryAyah — 200 clips
Clean studio clips from three reciters with zero samples in our training data (sahl_yassin, akram_alalaqimy, muhsin_al_qasim).
📖 QUL (unseen reciter) — 200 clips
Clean studio clips from Al-Nufais, a reciter we never trained on.
Method
- Decoding: greedy CTC (our/NeMo models) or each model's default decoder; Arabic auto-detected or forced per model.
- Mode: offline = full-utterance (sees the whole clip); streaming = true cache-aware, frame-by-frame. This reflects the model's actual encoder config, not its README — some repos claim streaming but ship offline weights.
- Metric: WER & CER over normalized text (diacritics/tatweel stripped; hamza/alef-maqsura unified). The alef-insensitive WER additionally strips alef to neutralize the rasm↔imlaei orthography gap, reflecting true word accuracy.
- PER (phoneme models): native phoneme error rate against the canonical phoneme string, at the letter-group unit level. Where a model has measured per-subset PER, the WER view shows it directly in the subset cells, marked "(PER)" — those cells are the model's raw phoneme accuracy, not retrieval-snapped WER. Overall/rank remains retrieval WER, flagged with *.
- Leakage control: the EveryAyah test split overlaps our training, so we report only reciters with zero training samples, plus a held-out QUL reciter and held-out real-phone tlog. nvidia FastConformer (current public #1) runs on the same held-out clips for a fair head-to-head.
- Field models (whisper, seamless, mms, cohere, omniASR…) are the best of each family from the Open Universal Arabic ASR Leaderboard. They are strong on general Arabic but degrade sharply on Quranic recitation (classical diacritized Arabic, distinct prosody) — which is the point of a Quran-specific benchmark.
- Tarteel official: all 600 clips were run through Tarteel's production realtime ASR at voice-v2.tarteel.io and scored with the official scorer.
- Not shown: Qwen3-Omni-30B and gemma-4-31B / 26B-A4B (too large for a 24 GB GPU, or vision-text not audio); obadx/muaalem (a phoneme/sifat model, not text ASR, so not WER-comparable).
🚀 Submit your model
The full benchmark is open, behind a manual-approval gate:
Quran-Lab/quranic-asr-benchmark —
600 held-out clips, references, and the official score.py (same normalization as this board).
- Request access to the dataset (approved per request).
- Transcribe every clip in
benchmark.jsonl(theaudiocolumn); write one hypothesis perid. - Run
python score.py --hyps my_hyps.jsonlto get per-source and overall WER/CER. - Open a discussion on this Space with your numbers or model link, and we'll add the row.
📉 Pareto frontier: WER vs model size
Models on the Pareto frontier achieve the best trade-off between accuracy and model size: for each frontier model, no smaller model achieves a lower Overall WER.
Names are shown for frontier models; hover over the other points to see their model name, family, size, mode, and WER.
@misc{quranlab2026benchmark,
title = {Quranic ASR Benchmark: a leakage-free held-out evaluation of Arabic ASR on Quranic recitation},
author = {{Quran-Lab}},
year = {2026},
howpublished = {\url{https://huggingface.co/datasets/Quran-Lab/quranic-asr-benchmark}}
}