Tushe/shami-tts-multi

Shami-TTS Multi — 10-Speaker Levantine ↔ English Code-Switching TTS

One model, ten voices, one shared phonetic space. Shami-TTS Multi extends the published Shami-TTS (single-voice) to 10 speakers (5 male / 5 female) with no engine hand-off at code-switch boundaries: all dialect and switch intelligence lives in a deterministic, unit-tested text→IPA front-end, and a compact 38.7M-parameter non-autoregressive VITS does waveform generation at 24 kHz.

• Code + training + eval scripts: github.com/Al-aminI/hams-levantine-tts
• Paper (single-speaker system): shami_tts.pdf in the repo above
• Inputs are phonemes from the shared IPA inventory (hams_tts.text.frontend), not raw text — use the front-end to convert text → phoneme/language IDs.

---

Measured on a consumer RTX 3060 12 GB (this exact card, not a datacenter GPU)

Metric	Target	Measured	Status
Peak VRAM (acoustic model, fp32)	≤ 3072 MB	224 MB	✅
TTFA p50 / p95 (streaming)	< 300 ms	46 / 199 ms	✅
RTF mean / p95 (HiFi-GAN decoder)	< 0.3	0.033 / 0.111	✅
Full production path (VITS→mel→BigVGAN→trim) RTF	—	0.105 (p95 0.159)	—
VRAM, VITS + BigVGAN resident	—	814 MB	—
Concurrent real-time streams (RTF 0.033, sequential)	—	~30 / GPU	—

Benchmark: benchmark.json in this repo (15 measurements, 3 prompts × 5 runs, warmup 2).

---

Quality — and the full improvement/ablation story

All numbers: held-out 76-utterance eval (all 10 speakers, 24 kHz), Whisper large-v3 ASR round-trip, auto-calibrated length scale. Levantine CER is inflated by Whisper's MSA bias — relative improvements are the valid claim.

Training stage	Pure-Lev CER	Code-switch CER	Overall CER	Dur. mean ratio @ls=1	Worst utt.	Length-corr	mel plateau
v3 — naive multi-speaker bolt-on (random cond init, c_dur 1.0)	0.173	0.521	0.356	0.933	0.744	−0.244	21–25
v3.1 — zero-init cond + c_dur 2.0 + long-utt ×3 oversampling	0.188	0.572	0.390	1.003	0.867	−0.236	21–25
v3.2 — sil/brk pause tokens + espeak ᵻ fix + lr-1e-4 polish	0.164	0.503	0.342	0.928*	0.733	−0.160	19.5–21.6

* the sentence-final brk token reserves ~7% of total duration for the utterance-final pause; synthesize at length_scale ≈ 1.08 for exact mean duration (auto-calibration built into scripts/eval_checkpoint.py --auto-ls).

Per-speaker CER (v3.2) — the female voices lead

Speaker	CER		Speaker	CER
spk_07_female	0.280		spk_01_male	0.270
spk_08_female	0.313		spk_04_male	0.285
spk_09_female	0.346		spk_02_male	0.309
spk_10_female	0.350		spk_03_male	0.423
spk_06_female	0.372		spk_05_male	0.487
female avg	0.332		male avg	0.353

What each stage taught us (the engineering log)

1. Naive conditioning destabilizes a trained VITS (v3). Bolting a randomly-initialized speaker-conditioning layer-set onto the posterior encoder / flow / decoder / duration predictor of a warm-started single-speaker model cost −8% duration calibration and an elevated mel plateau.
2. Zero-init conditioning is the clean bolt-on (v3.1). Zero the scale of every conditioning conv and the warm-start is a bit-exact behavioural clone of the single-speaker model at step 0 (proved: max output diff 0.0, same noise seed). Conditioning then grows under the GAN+mel gradients. Duration mean went 0.933 → 1.003; worst utterance 0.744 → 0.867.
3. Duration loss weight matters: the proven deterministic-duration recipe (c_dur 2.0) had drifted to 1.0 in v3 — restoring it was part of the v3.1 fix.
4. Long utterances need oversampling (v3.1). Training clips are p50 3.7 s; the duration head under-predicted increasingly with length (corr −0.29). Oversampling every ≥6 s clip ×3 halved the worst-case shortfall.
5. Pauses must be first-class symbols (v3.2). The reference speech carries deliberate phrase pauses (measured: 45 pauses / median 200 ms per 10 eval clips) that our old stream could not represent — punctuation was stripped before G2P, so MAS dumped silence onto neighbouring phonemes. The fix: dedicated sil (comma) / brk (sentence) symbols, appended to the frozen inventory (88 → 90), emitted by the front-end at punctuation, with an embedding-resize warm start (new rows zeroed). On punctuated input the model now emits real 170–380 ms pauses.
6. One-character G2P bugs produce audible garbage (v3.2). espeak's reduced-vowel symbol ᵻ ("delay", "believe") became a literal <unk> string that char-split into garbage tokens. Mapped ᵻ → ə (schwa) + fold-table defense.
7. torch weight-norm parametrization roles (v3.1): original0 = scale g, original1 = direction v; computed weight g·v/‖v‖. Fresh HF-built cond layers materialize with both buffers zero → 0/0 = NaN from step 0 (posterior/flow poisoned, duration path finite — the diagnostic tell). Zero the scale, never the direction. Guard ships in hams_vits.py.

Honest limitations

• Interior pauses are only produced where text marks them. The source corpus's own phrase pauses were inserted by the source TTS at points the transcripts never marked (commas exist in only 2.6% of training rows) — unmarked pauses are information-theoretically absent from text. Punctuate your input for phrased output.
• Residual length-sensitivity (corr −0.16): very long utterances still run slightly fast — largely inherent to the compact 192-hidden encoder.
• The smoothness ceiling is the architecture: a 36M VITS distilling synthetic 24 kHz speech. Multi-speaker mel plateaus ~19.5 vs ~17.8 single-speaker on the same lineage.
• Whisper's Arabic ASR is MSA-biased; absolute CERs are inflated — compare relatively.

---

Samples (held-out utterances: reference / HiFi-GAN decoder / VITS→BigVGAN)

Female voices

spk_07_female (CER 0.280)

text	REF	HiFi-GAN	BigVGAN
held-out #1
held-out #2

spk_08_female (CER 0.313)

text	REF	HiFi-GAN	BigVGAN
held-out #1
held-out #2

spk_09_female (CER 0.346) — incl. novel pure-Levantine / English / code-switched

text	HiFi-GAN	BigVGAN
held-out #1
held-out #2
novel · pure Levantine
novel · pure English
novel · code-switched

spk_10_female (CER 0.350)

text	REF	HiFi-GAN	BigVGAN
held-out #1
held-out #2

spk_06_female (CER 0.372)

text	REF	HiFi-GAN	BigVGAN
held-out #1
held-out #2

Male voices

spk_01_male (CER 0.270) — incl. novel sentences

text	REF	HiFi-GAN	BigVGAN
held-out #1
held-out #2
novel · code-switched	—

spk_02_male (CER 0.309): ref/hifi/bigvgan · spk_04_male (0.285): ref/hifi/bigvgan · spk_03_male (0.423): ref/hifi/bigvgan · spk_05_male (0.487): ref/hifi/bigvgan

---

Usage

import sys, torch, soundfile as sf
sys.path.insert(0, "src")  # github repo: Al-aminI/hams-levantine-tts

from hams_tts.models.hams_vits import HamsVITS
from hams_tts.text.frontend import TextFrontend
from hams_tts.inference.bigvgan_vocoder import synthesize

model = HamsVITS.from_checkpoint("checkpoints/shami-tts-multi").cuda().eval()
fe = TextFrontend()

utt = fe.process("بكرا عندي meeting مع ال client, بس ال flight لت delay ساعتين.")
spk_id = model.speaker_map["spk_09_female"]        # 10 voices, ids persisted in the config

# acoustic path (224 MB VRAM, RTF 0.033)
wav = model.infer(
    torch.tensor([utt.phoneme_ids], device="cuda"),
    torch.tensor([utt.language_ids], device="cuda"),
    speaker_id=torch.tensor([spk_id], device="cuda"),
    length_scale=1.08,                              # brk reserves ~7% for the final pause
).squeeze().cpu().numpy()

# or the production path incl. BigVGAN + trailing-artifact trim (814 MB, RTF 0.105)
wav = synthesize(model, utt.phoneme_ids, utt.language_ids,
                 length_scale=1.08, speaker_id=spk_id)

sf.write("out.wav", wav, model.sample_rate)        # 24 kHz

Speaker ids: spk_01_male=0 … spk_10_female=9 (persisted in hams_vits_config.json: speaker_map).

---

Training details

• Data: lahgtna-levantine-tts (CC-BY-4.0) — 59.5 h / 47,945 filtered clips / 10 speakers (5M/5F), 24 kHz, ~12% code-switched, diacritized + de-desinentialized via the deterministic front-end (camel-tools + dialectal rules).
• Hardware: single NVIDIA GeForce RTX 3060 12 GB (Ampere, bf16).
• Multi-speaker adaptation: ~12.8 GPU-hours across three runs (v3 30k steps ≈ 5.0 h incl. two crash-resumes via atomic checkpointing; v3.1 30k ≈ 5.1 h; v3.2 16k ≈ 2.7 h), warm-started from the single-speaker flagship (v4_tex, itself ~26k steps of lineage).
• Recipe: deterministic duration head (MSE on MAS log-durations), c_dur 2.0, mel×45, LSGAN MPD/MSD + feature-matching c_fm 4.0, KL, persisted discriminator across runs, batch 12, seg 12288, AdamW 2e-4 → 1e-4 (polish), long-utterance (≥6 s) ×3 oversampling, sil/brk pause tokens in the phoneme stream.
• Best checkpoints by eval sweep (GAN losses are non-monotonic — always sweep): v3.2 step 8000 promoted here.

Attribution & license

• Base acoustic model: facebook/mms-tts-ara (CC-BY-NC 4.0) → this model is cc-by-nc-4.0.
• Training data: lahgtna (CC-BY-4.0). Vocoder option: nvidia/bigvgan_v2_24khz_100band_256x.
• By the Tushe Language Research Team. Single-speaker system + paper: Tushe/shami-tts.