Turkish speech recognition: model, CPU inference (offline and streaming) and training code. Experimental Moroccan Darija
(balbal-v2e-zellige) and Ancient Greek (balbal-v2e-calliope) models run through the same balbal package.
| Model | Size | Encoder | Tokenizer | Languages |
|---|---|---|---|---|
| balbal-v2 (default) | 45.4M parameters (30.0M of them a product-key memory table) | Conformer-CTC, 16 layers, d_model 176, 4x subsampling | SentencePiece 1024 | Turkish |
| balbal-v2-manas (large) | 313.3M parameters (192.0M of them a product-key memory table) | FastConformer-CTC, 18 layers, d_model 512, 4x subsampling | SentencePiece 2048 | Turkish, English |
| balbal-v2e-zellige (experimental) | 307.3M parameters (193M of them a product-key memory table) | FastConformer-CTC, 17 layers, d_model 512, 8x subsampling, initialised from Sabeq-ASR | SentencePiece 1289 | Moroccan Darija |
| balbal-v2e-calliope (experimental) | 314.4M parameters (194M of them a product-key memory table) | FastConformer-CTC, 18 layers, d_model 512, 4x subsampling, fine-tuned from balbal-v2-manas on 100% synthetic speech | SentencePiece 3068 | Ancient Greek |
All four models add a 3-pass loop refiner with a product-key acoustic memory on top of the CTC encoder and decode with a 6-gram token KenLM beam search. Output: cased, punctuated text with per-word timestamps (balbal-v2e-zellige: Arabic-script text as decoded, offline only; its weights are CC BY-NC-SA 4.0, non-commercial; balbal-v2e-calliope: lowercase polytonic Greek without punctuation).
This repository holds the code; the weights are on Hugging Face. Benchmarks, a browser demo and the Android app are on the model page (Türkçe). The previous model's code is in balbal-v1/.
pip install balbal # Linux x86-64, macOS (Apple Silicon), Windows x86-64; Python 3.11 or 3.12
balbal run speech.wav # balbal-v2; the model downloads from Hugging Face on the first run
balbal run --model manas --json speech.mp3 # balbal-v2-manas; text + per-word timestamps and confidences
balbal run --model zellige speech.wav # balbal-v2e-zellige: Moroccan Darija (experimental, non-commercial weights)
balbal run --model calliope speech.wav # balbal-v2e-calliope: Ancient Greek, classical pronunciation (experimental)
balbal run -f srt -o subs/ videos/ # subtitles for every audio / video file in a folder
pip install 'balbal[live]'; balbal live # live microphone transcription
pip install 'balbal[server]'; balbal serve # OpenAI-compatible API + live WebSocket
pip install 'balbal[wyoming]'; balbal wyoming # speech-to-text for Home Assistantimport balbal
print(balbal.transcribe("speech.wav")["text"])On Android: balbal-android-0.9.apk (36 MB, Android 8+, arm64). Both models run offline on the phone; tap İndir in the app to download one. The big button records and transcribes once; Canlı shows the text while you speak.
The balbal package is the CPU inference below (fast graph + native KenLM beam decoder),
prebuilt; its sources and wheel build are in pypi/. The rest of this README runs from a checkout.
balbal/ shared Python package: model definitions, model builder, audio reading, text normalizer and scorer,
per-language rules (lang.py)
configs/ balbal-v2/ and balbal-v2-manas/: NeMo model config, refiner + memory config, decoder settings
inference/
transcribe.py offline transcription (runtimes fast / onnx / torch)
runtime.py the three runtimes + model directory layout
decoder.py KenLM beam decoder (+ hotword biasing, word timestamps)
export_onnx.py PyTorch weights -> the two CPU ONNX graphs (+ onnx_tools.py)
build_native.py builds the native decoders in native/ (C++ / pybind11)
kernels/ onnxruntime custom operators of the fast graph (C++ sources + build scripts, x86-64 and Android arm64)
streaming/ streaming recognizer, WebSocket server, file client, browser microphone demo, exactness test
training/
balbal_v2/train.py, manas/train.py trainers (shared loop and data code in common/)
soup.py weight soups
build_lm.py 6-gram token KenLM for the decoder
requirements/ cpu.txt (inference without torch), full.txt (export, torch runtime, training)
pypi/ the pip package (balbal run / live / serve / wyoming; Hugging Face download), its tests and manylinux wheel build
balbal-v1/ balbal-v1 code, kept for reproducibility
Python 3.12 on Linux x86-64 (macOS and Windows: see the notes after the build commands).
# CPU inference only (runtimes fast + onnx, streaming)
pip install -r requirements/cpu.txt
# or everything (ONNX export, runtime torch, training): torch 2.10.0 + NeMo 2.5.0
pip install torch==2.10.0 torchaudio==2.10.0 --index-url https://download.pytorch.org/whl/cu128 # or a ROCm / CPU build
pip install --no-deps -r requirements/full.txtBuild the native decoders (needs a C++17 compiler; flashlight-text==0.0.7 and pybind11==2.10.4 from the requirements):
python inference/build_native.py # builds inference/native/*.so, then checks them against flashlight-textBuild the custom operators of the fast graph (needs g++; the onnxruntime 1.30 headers are vendored in inference/kernels/third_party/):
inference/kernels/build.sh # -> inference/kernels/libbalbal_ops.so (MARCH=x86-64-v3 for a portable build)
python inference/kernels/test_kernels.py # PkmRead / PkmReadQ8 vs numpy, RelPosScore vs the stock operators (bit-identical)
ANDROID_NDK=/path/to/ndk inference/kernels/build_android.sh # Android arm64 -> inference/kernels/android-arm64/BALBAL_OPS_LIB points the runtime at another build of the library.
On macOS (Apple Silicon) the same two scripts build with the Xcode Command Line Tools (xcode-select --install):
build_native.py writes inference/native/*.so, build.sh writes inference/kernels/libbalbal_ops.dylib, and the runtime
and test_kernels.py load the .dylib.
On Windows, build_native.py compiles with MSVC (Visual Studio Build Tools, "Desktop development with C++"; flashlight-text's
Windows wheel is an MSVC build, so the binding ABI must match) and writes inference/native/*.pyd, plus kenlm.dll, built from
KenLM: flashlight-text's Windows KenLM module needs it, but its wheel does not include it. build.sh runs in Git Bash
with LLVM's clang++ and writes inference/kernels/balbal_ops.dll. Training on Windows: training/WINDOWS.md.
The PyPI wheels (Linux x86-64, macOS arm64, Windows x86-64) are built by pypi/build_wheels.sh (manylinux) and
pypi/build_wheel.py (macOS, Windows).
Every tool takes --model-dir MODEL_DIR (or the environment variable BALBAL_MODEL_DIR):
MODEL_DIR/
model.json {"model": "balbal-v2"} or {"model": "balbal-v2-manas"} (+ optional fields, see below)
model.pt PyTorch weights (runtime torch, ONNX export)
refiner.json, model_config.yaml optional: this model's architecture, replacing configs/<family>/ (other memory size, vocabulary)
decoder.json optional: this model's KenLM decoder settings, replacing configs/<family>/decoder.json
tokenizer/ tokenizer.model, tokenizer.vocab, vocab.txt
lm/ model.binary (KenLM), tokenizer.model
onnx/model.onnx (+ .data) fast graph, custom operators (runtime fast, the default)
onnx/model-plain.onnx (+ .data) standard operators only (runtime onnx)
The ONNX graphs are made from model.pt (any 16 kHz-readable clips serve as the parity check; a few seconds each):
python inference/export_onnx.py --model-dir MODEL_DIR clip1.wav clip2.wav clip3.wavIt writes both graphs and onnx/export.json (log-probability and frame-argmax agreement with PyTorch). balbal-v2-manas stores its
matrix weights in int8 by default (--int8 on|off); both fast graphs store the memory table in int8.
model.json fields (only model is required):
| Field | Meaning | Default |
|---|---|---|
model |
architecture family: balbal-v2 or balbal-v2-manas |
balbal-v2 |
name |
display name (results, servers) | the family |
languages |
spoken languages, primary first, e.g. ["az"]; the primary language picks the number writing (ITN), the hotword casing and the scoring rules (balbal/lang.py) |
["tr"] / ["tr", "en"] |
itn |
the language whose ITN transcribe() applies, or false |
the primary language's (only Turkish has one) |
long_windows |
[window, overlap] seconds for audio longer than 30 s |
[30, 10] |
python inference/transcribe.py --model-dir MODEL_DIR speech.wav # fast CPU runtime
python inference/transcribe.py --model-dir MODEL_DIR --runtime onnx speech.wav # standard onnxruntime operators only
python inference/transcribe.py --model-dir MODEL_DIR --runtime torch speech.wav # PyTorch / NeMo (--device cuda for a GPU)
python inference/transcribe.py --model-dir MODEL_DIR --hotwords "Eray,Hasan" call.wavOne JSON line per file: text, words (word, start, end in seconds), duration_seconds, inference_seconds,
realtime_factor. --text-only prints the text alone, --threads sets the CPU threads (default ASR_THREADS or 8),
--decoder flashlight runs flashlight-text's own beam search (same best path as the default native decoder). Numbers come out
as digits (bin dokuz yüz doksan beş -> 1995, a number run is one timed word); --no-itn (or itn=False) keeps the spoken
words the model writes. Audio longer than 30 s is read in overlapping windows whose CTC frames are
stitched and decoded in one KenLM pass (no word is cut at a window edge). WAV / FLAC / OGG are read directly, other formats through FFmpeg.
Domain LM. For words the general KenLM rarely sees (medicine, law, product names), build a small KenLM on your own text and mix it in:
python training/build_lm.py --tokenizer MODEL_DIR/lm/tokenizer.model --text my-sentences.txt --heldout my-test.jsonl --out my-lm
python inference/transcribe.py --model-dir MODEL_DIR --domain-lm my-lm --domain-weight 0.3 dictation.wavThe decoder mixes the two linearly, p = (1 - w) p_general + w p_domain, so a token missing from your text keeps the general
LM's probability. --text takes plain text (one sentence per line), --train transcript manifests; --heldout keeps your test
sentences out of the LM. Tune the weight (default 0.3) on recordings whose text is not in the domain LM. The same settings in a
model folder's decoder.json ("domain_lm": "my-lm", relative to the folder, and "domain_weight": 0.3) apply to
transcribe.py, the streaming server and the pip package. Decoding takes about 10% longer (the acoustic model's time is
unchanged).
In Python:
import sys; sys.path[:0] = ['inference', '.']
from runtime import Engine
from balbal.audio import read_audio
engine = Engine('MODEL_DIR') # runtime='fast' | 'onnx' | 'torch'
print(engine.transcribe(read_audio('speech.wav'))['text'])The models are full-context, so streaming re-runs the graph on recent audio. Two modes:
segment(default): every 0.32 s the graph runs once over the open segment and the whole segment is re-decoded. Partials equal the final text except for the last ~0.6 s. A segment ends after 0.5 s of quiet, blank audio, or at 5 s (cut at a short pause first).window: a sliding window (8 s left context, 0.32 s chunk, 0.64 s lookahead) commits each chunk once to an incremental beam search; a closed segment is re-decoded with full context for its final text.
The streaming decoder (inference/native/ctc_stream_decoder.cpp) returns the same best path as the offline decoder on the same frames.
python inference/streaming/server.py --model-dir MODEL_DIR [--mode segment|window] [--port 8791] [--workers 2 --threads 4]
python inference/streaming/client.py speech.wav --url ws://127.0.0.1:8791/ws # streams a file in real time
# browser microphone demo: http://127.0.0.1:8791/
python inference/streaming/test_equiv.py --model-dir MODEL_DIR a.wav b.wav # streaming == offline checksWebSocket protocol (ws://HOST:PORT/ws):
| Direction | Message | Content |
|---|---|---|
| client → server | binary | little-endian int16 PCM, mono, 16 kHz, any piece size |
| client → server | text {"type": "end"} |
no more audio |
| server → client | ready |
model, config, worker |
| server → client | partial |
text (open segment incl. the provisional tail), stable, start, joins, audio_end |
| server → client | final |
text, start, end, joins (the segment continues the previous word) |
| server → client | done |
transcript, stats |
In Python:
import sys; sys.path[:0] = ['inference/streaming']
from streamer import Engine, SegmentStream
engine = Engine('MODEL_DIR') # one per process
stream = SegmentStream(engine) # one per live stream (Stream = window mode)
for piece in pieces: events = stream.accept(piece) # float32, 16 kHz
events += stream.finish(); print(stream.transcript())Manifests are JSON lines, one row per clip:
{"id": "a1", "audio_path": "clips/a1.wav", "duration_seconds": 4.2, "text": "Merhaba, nasılsınız?", "source": "my-set"}Optional fields: lang (tr, en or ns for non-speech; default --default-lang, ns for an empty text), lang_spans
([[start_s, end_s, "en"], ...] for code-switching), pcm_sha256 (checked when present), parts (a row joined from segments of
other files). See training/common/data.py.
# balbal-v2: from balbal-v2 (or balbal-v1) weights
python training/balbal_v2/train.py runs/my-balbal --tokenizer-dir MODEL_DIR/tokenizer --init MODEL_DIR/model.pt \
--train train.jsonl --dev dev.jsonl
# balbal-v2-manas: from balbal-v2-manas weights, here with the fine-tune settings
python training/manas/train.py runs/my-manas --tokenizer-dir MODEL_DIR/tokenizer --init MODEL_DIR/model.pt \
--train train.jsonl --dev dev.jsonl --lr 5e-5 --epochs 2 --decay .5 --soup ""
# balbal-v2-manas from NVIDIA's English FastConformer-CTC-Large (encoder only; head, refiner and memory start fresh)
python training/manas/train.py runs/my-manas --tokenizer-dir MY_TOKENIZER --nemo stt_en_fastconformer_ctc_large.nemo --step-offset 0 \
--train train.jsonl --dev dev.jsonl
# multi-GPU
torchrun --nproc_per_node 4 training/balbal_v2/train.py runs/my-balbal ...The defaults are the release recipes:
| balbal-v2 | balbal-v2-manas | |
|---|---|---|
| Update | 64 clips, micro-batch 16 | 64 clips, micro-batch 8 |
| Length | 5 epochs | 4.5 epochs, then a 2-epoch fine-tune |
| LR | 1.4e-4, 500 warmup updates, flat, linear decay over the last 20 % to 10 % | 1.4e-4, 500 warmup updates, flat to epoch 2.5, linear decay to 10 %; fine-tune 5e-5, decay over the last 50 % |
| Optimizer | Muon + AdamW, weight decay .001 | Muon (bf16 Newton-Schulz) + AdamW, weight decay .01 |
| Memory table LR | 5x | 5x (and the learned read temperature) |
| Objective | 0.3 x base CTC + 0.7 x mean CTC of the 3 refiner passes + 0.1 x frame language CE | same |
| Released weights | final checkpoint of the 5 epochs | uniform soup of the fine-tune's 7 evaluations, epochs 0.5 to 2.0 |
Each run writes config.json, its architecture files (refiner.json, model_config.yaml, model.json), metrics.jsonl (a progress line every 100 updates, an evaluation every quarter epoch with greedy WER /
CER on --dev, overall and per source), the best checkpoints by dev WER in checkpoints/, state.pt (--resume continues the
run) and final.pt. --cond also trains an utterance-level language condition (tr / en / auto). All options: --help.
python training/soup.py soup.pt runs/my-manas/checkpoints/step*.pt # uniform weight soup
python training/build_lm.py --tokenizer MODEL_DIR/tokenizer/tokenizer.model --train train.jsonl --heldout dev.jsonl \
--out MODEL_DIR/lm --quantize # the decoder's 6-gram token KenLMThe models and the trainers are not tied to Turkish. A model for another language needs its own tokenizer, training data and KenLM; the rest is configuration:
- Tokenizer: a SentencePiece model in
tokenizer/(tokenizer.model,tokenizer.vocab,vocab.txt). The CTC head takes its size from it. - Architecture files: a folder with a copy of
configs/<family>/refiner.json, whereconfig.vocabularyis the tokenizer size,languagesis the language head's languages (e.g.["az"]; non-speech is added after them) and the memory may be resized (memory.subkeys). Pass it as--config-dir; amodel_config.yamlthere replaces the family's too. - Manifests:
langis one of those languages orns;--default-langdefaults to the first one. Dev WER / CER follow that language's scoring rules (balbal/lang.py: Turkish has its own normalizer, other languages are scored lowercased without punctuation). - Training: e.g. from NVIDIA's English encoder:
The run directory gets
python training/manas/train.py runs/my-az --tokenizer-dir MY_TOKENIZER --config-dir MY_CONFIG \ --nemo stt_en_fastconformer_ctc_large.nemo --step-offset 0 --train train.jsonl --dev dev.jsonlrefiner.json,model_config.yamlandmodel.json({"model": "balbal-v2-manas", "languages": ["az"]}).--initfrom weights with another language head size keeps everything else and starts the head fresh. - KenLM:
python training/build_lm.py ... --language az(held-out sentences are matched with that language's text key). - Model folder: the checkpoint as
model.pt,tokenizer/,lm/and the run'srefiner.json/model_config.yaml/model.json(add"name", anddecoder.jsonif you tuned alpha / beta), thenpython inference/export_onnx.py --model-dir MODEL_DIR clips....
Upload that folder (with onnx/) to Hugging Face and pip install balbal runs it: balbal.load('owner/name') or
balbal run --model owner/name speech.wav. The pip package needs model.json, lm/ and onnx/; decoder.json is read when present.
A language gets its own number writing or scoring by adding an entry to LANGUAGES in balbal/lang.py (an ITN like
balbal/normalizer/itn.py, a scorer like balbal/normalizer/metrics_v2.py). Without one, numbers stay as the model writes them.
The commands used to test this release on CPU (MODEL_DIR with model.pt, tokenizer/ and lm/; clip.wav a few seconds of Turkish speech):
python inference/build_native.py
inference/kernels/build.sh && python inference/kernels/test_kernels.py
python inference/export_onnx.py --model-dir MODEL_DIR clip.wav other1.wav other2.wav
for r in fast onnx torch; do python inference/transcribe.py --model-dir MODEL_DIR --runtime $r --text-only clip.wav; done # same text
python inference/streaming/test_equiv.py --model-dir MODEL_DIR clip.wav other1.wav other2.wav
python inference/streaming/server.py --model-dir MODEL_DIR --port 8791 & # then:
python inference/streaming/client.py clip.wav --url ws://127.0.0.1:8791/ws
LOADER_WORKERS=0 python training/balbal_v2/train.py runs/check --tokenizer-dir MODEL_DIR/tokenizer --init MODEL_DIR/model.pt \
--train toy.jsonl --dev toy-dev.jsonl --batch 4 --micro 2 --updates 3 --eval-epochs 1 --progress-every 1Issues and pull requests are welcome; see CONTRIBUTING.md.
Code: Apache License 2.0 (LICENSE, NOTICE). The balbal-v2-manas model configuration derives from NVIDIA's
stt_en_fastconformer_ctc_large (CC-BY-4.0). Weights: see their model cards (balbal-v2, balbal-v2-manas, balbal-v2e-zellige: CC BY-NC-SA 4.0, balbal-v2e-calliope). How to cite: CITATION.cff.