MustaqiLLM

MustaqiLLM is a 5.17-billion-parameter Uzbek chat and text-classification model. It follows Uzbek instructions reliably, writes fluent Uzbek in both Latin and Cyrillic script, and is strong on sentiment and news classification. It is not a knowledge model: on multiple-choice knowledge benchmarks it performs at chance. Read the Evaluation and Limitations sections before using it — they are specific about what works and what does not.

Parameters	5.17 B
Architecture	NeuronLMForCausalLM (custom, ships with the repo)
Layers / hidden	36 / 3584
Attention	GQA, 28 query heads : 4 KV heads, head_dim 128, QK-norm
Position encoding	RoPE, θ = 500000
Context length	4096 tokens
Vocabulary	48,000 (BPE)
Embeddings	untied
Weights dtype	bfloat16 (embeddings and lm_head stored fp32)
Languages	Uzbek (Latin + Cyrillic), English, Russian

---

Quick start

The architecture is custom, so trust_remote_code=True is required — the modeling code ships inside this repository.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NeuronUz/MustaqiLLM"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,       # weights are bf16; do not load in fp32
    device_map="cuda",
).eval()

messages = [{"role": "user", "content": "O'zbekistonning poytaxti qaysi shahar?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,           # greedy; see Generation settings below
        eos_token_id=5,            # <|im_end|> -- also the repo default
        pad_token_id=3,            # <pad>
    )

print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Oʻzbekistonning poytaxti - Toshkent.

Chat template

The model uses ChatML. tokenizer.apply_chat_template applies it for you; the raw form is:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>

A system turn is optional. Uzbek-language system prompts work best — that is what the model was trained with.

Generation settings

setting	value	why
eos_token_id	5 (<|im_end|>)	The turn terminator, and already the default in config.json / generation_config.json — you do not need to pass it. Do not override it with the pretraining EOS (</s>), which never appears in chat data: generation would then run to max_new_tokens.
do_sample	False for classification/extraction; True, temperature≈0.7, top_p≈0.9 for open chat	generation_config.json ships do_sample: true with no temperature or top_p set, so pass these explicitly. Every benchmark below was measured greedy.
dtype	torch.bfloat16	Trained in bf16.

Memory: the checkpoint is 11.0 GB on disk (embeddings and lm_head are stored fp32); loading with dtype=torch.bfloat16 as above casts them down to ~10.3 GB of weights, so a single 16 GB GPU is enough for inference.

config.json sets use_cache: false, but generation_config.json sets use_cache: true, so generate() uses the KV cache. Pass use_cache=True explicitly if you write your own decode loop.

Serving

vLLM and SGLang cannot load this model. They reimplement each architecture internally rather than executing a repository's Python, and NeuronLMForCausalLM is not in their model registries — trust_remote_code only covers the config and tokenizer there. Use the transformers backend, or convert the weights (the architecture is Qwen3-equivalent apart from fused qkv_proj / gate_up_proj and out_proj naming; splitting those tensors and renaming to the Qwen3 layout yields a checkpoint vLLM will serve).

---

Training

MustaqiLLM is the instruction-following stage on top of an in-house 5.17 B Uzbek pretrained model. The pretrained checkpoint completed a single pass over its corpus; this stage is supervised fine-tuning only — no continued pretraining was performed.

Method	Full-parameter SFT (no LoRA)
Data	190,939 instruction/chat examples
Epochs	3 (17,701 steps); released checkpoint is from epoch 2.75
Effective batch	32 (micro-batch 8 × grad-accum 4)
Sequence length	2048
Optimizer	AdamW fused, β₁ 0.9, β₂ 0.95, weight decay 0.0
LR schedule	1e-5, cosine, 3% warmup, grad-norm clip 1.0
Precision	bf16 mixed precision, gradient checkpointing
Hardware	1 × NVIDIA RTX PRO 6000 Blackwell (96 GB), ~10 h

Data composition

slice	rows	share
Uzbek instruction/chat backbone (curated + filtered)	169,919	89.0%
Uzbek Cyrillic chat (transliterated)	12,000	6.3%
Russian-instructed translation	5,000	2.6%
Latin ↔ Cyrillic script conversion	3,000	1.6%
Cyrillic identity/social	1,020	0.5%

The backbone mixes general Uzbek assistant data, benchmark-format task data (MCQ, classification, spelling), English↔Uzbek translation pairs, and an English retention slice. Third-person rubric-grading text was filtered out of the backbone before training.

The Cyrillic and Russian slices exist because the pretrained model reads and writes Uzbek Cyrillic better than Latin (bits-per-byte 0.2288 vs 0.3868) yet had almost no Cyrillic chat behaviour attached to it, and because Russian-language instructions were nearly absent. Checkpoint selection was done by running the full benchmark suite on all 12 saved checkpoints, not by held-out loss — held-out loss was flat (1.784–1.796) across the last two epochs while benchmark scores were still moving.

---

Evaluation

Full public benchmark suite, greedy decoding, transformers backend, seed 42, complete test sets (no subsampling). Scores are accuracy unless noted.

Uzbek benchmarks

benchmark	n	score	invalid rate
uzlib (Uzbek linguistic MCQ)	1,861	0.2875	0.0000
TUMLU-Uzbek (Uzbek MMLU)	700	0.3286	0.0000
MMLU-Uz (translated MMLU)	14,042	0.2584	0.0000
News topic classification (10-way, risqaliyevds/uzbek-zero-shot-classification)	96,970	0.6531	0.0000
Sentiment (binary)	10,000	0.9259	0.0001

Random baselines: 0.25 for the 4-way MCQ tasks, 0.10 for news, 0.50 for sentiment.

English

benchmark	n	score	invalid rate
MMLU (English)	14,042	0.2619	0.0000

Translation (FLORES+)

direction	n	BLEU	COMET	length ratio
English → Uzbek	2,009	5.17	0.7397	1.018
Uzbek → English	2,009	1.83	0.5376	1.229

uzlib, per split

split	n	score
fill_in	52	0.3077
correct_word (orthography)	1,501	0.3011
meaning_in_context	72	0.2639
meaning	236	0.2034

News, per class

class	n	score
Sport	16,113	0.8743
Texnologiya (Technology)	5,177	0.7309
Madaniyat (Culture)	2,405	0.7081
Siyosat (Politics)	29,500	0.6794
Iqtisodiyot (Economy)	10,755	0.6596
Salomatlik (Health)	3,505	0.6579
Ta'lim (Education)	1,987	0.6548
Ekologiya (Ecology)	1,784	0.5667
Xorijiy Yangiliklar (World news)	11,732	0.5124
Oila va Jamiyat (Family & Society)	14,012	0.4273

Limitations

Multiple-choice knowledge tasks perform at chance. uzlib, MMLU-Uz and MMLU-English all sit within noise of their 0.25 random baseline, across roughly 30,000 questions. Invalid rates near zero mean the model answers in the correct format every time and is still wrong — this is missing knowledge, not broken parsing. The underlying pretrained model completed a single pretraining epoch, and supervised fine-tuning cannot add facts that were never learned. Do not use this model for factual question answering, exams, or retrieval-free knowledge tasks. TUMLU-Uzbek at 0.3286 is the only MCQ result above chance, and its 700-item sample gives it a ±3.5% confidence interval.

Uzbek → English translation is weak. BLEU 1.83 with a 1.229 length ratio and 12.5% unigram precision means the model over-generates English that mostly does not match the reference. English → Uzbek is usable (COMET 0.7397) but not competitive with dedicated translation systems.

Script conversion does not work despite being trained for it. Asked to transliterate Latin Uzbek to Cyrillic, the model frequently returns the input unchanged. The 3,000-row slice was too small.

The Cyrillic slice was machine-transliterated, and its artifacts are visible in output. Loanwords and brand names inside Cyrillic text can come out mangled (e.g. Facebook → Факебоок), and occasional single Cyrillic characters leak into Latin words. Cyrillic chat is coherent and does not degenerate, but Cyrillic orthography is less reliable than Latin.

Self-identification. The identity training data predates the current name, so asked who it is, the model answers "NeuronAI 5B" rather than "MustaqiLLM".

News classification is uneven. The "Oila va Jamiyat" (Family & Society) class scores 0.4273 across 14,012 items — a semantically diffuse catch-all the model handles poorly, against 0.8743 for the lexically distinctive Sport class.

Safety. No safety alignment, RLHF, or red-teaming was performed. The model has no refusal training beyond what the instruction data incidentally contains. It can produce incorrect, biased, or unsafe content, and — given the benchmark results above — will state false facts fluently and confidently. Evaluate it for your own use case before deploying it anywhere user-facing.

---

Intended use

Suitable for: Uzbek-language chat and assistance; text classification (sentiment, topic); Uzbek text generation and rewriting in Latin or Cyrillic; English → Uzbek translation where approximate meaning suffices; a base for further fine-tuning.

Not suitable for: factual question answering or anything knowledge-intensive; exam-style multiple choice; Uzbek → English translation; script transliteration; any application where a confidently-stated wrong fact causes harm (medical, legal, financial advice).

License

Apache 2.0. Training data licensing follows the sources of the underlying public datasets.

Citation

@misc{mustaqillm,
  title  = {MustaqiLLM: an instruction-tuned Uzbek language model},
  author = {NeuronUz},
  year   = {2026},
  url    = {https://huggingface.co/NeuronUz/MustaqiLLM}
}