ArmGPT (110M)

A GPT language model trained on Armenian text data.

Model Details

Property	Value
Parameters	109,923,072 (110M)
Architecture	Transformer (RMSNorm, SwiGLU, RoPE)
Layers	12
Heads	12
Embedding dim	768
Context length	512 tokens
Vocab size	16,000
Tokenizer	BPE (SentencePiece)
Training steps	24,000

Training Data

Trained on Armenian text from multiple sources:

• Armenian Wikipedia
• CC-100 Armenian
• CulturaX Armenian
• OSCAR Armenian
• HPLT Armenian
• Glot500 Armenian

Usage

import torch
import json
from core.model import GPT
from core.bpe_tokenizer import BPETokenizer

# Load model
checkpoint = torch.load("model.pt", map_location="cpu")
with open("config.json") as f:
    cfg = json.load(f)

tokenizer = BPETokenizer.load("tokenizer.json")

model = GPT(
    vocab_size=cfg["vocab_size"],
    n_layer=cfg["n_layer"],
    n_head=cfg["n_head"],
    n_embd=cfg["n_embd"],
    block_size=cfg["block_size"],
    dropout=0.0,
)
model.load_state_dict(checkpoint)
model.eval()

# Generate text
prompt = "Հայաստան"
ids = tokenizer.encode(prompt)
context = torch.tensor([ids], dtype=torch.long)
output = model.generate(context, max_new_tokens=200, temperature=0.8, top_k=40)
print(tokenizer.decode(output[0].tolist()))

License

Apache 2.0