Nexus-Erebus-50M

Nexus-Erebus-50M

Nexus-Erebus-50M is a compact ~54M-parameter language model trained from scratch by Ideoa Labs, combining strong commonsense reasoning with genuine integer arithmetic ability at tiny scale.

Model details

Parameters ~54.1M
Architecture Llama-style decoder
Hidden size 512
Layers 9
Attention heads 8
Vocab size 32,000 (custom digit-aware tokenizer)
Context length 1,024
Precision bfloat16

The tokenizer keeps digits atomic rather than merging them into BPE units, which preserves the positional structure that integer arithmetic depends on.

Benchmarks

Measured with lm-eval-harness, 0-shot, acc_norm, on the full test sets (no subsampling). ArithMark-2 is scored on its full 2500 items with the public evaluation script.

Task Items Nexus-Erebus-50M
ARC-easy 2,376 44.40
ARC-challenge 1,172 22.70
HellaSwag 10,042 27.05
PIQA 1,838 58.27
ArithMark-2 2,500 52.48
Average 40.98

Against the sub-100M field

Published Open SLM Leaderboard values, same five tasks, same full-set protocol.

Model Params Average
Nexus-Erebus-50M 54M 40.98
Atom 2.7M 3M 40.43
Supra-1.5-50M-base-exp 52M 39.00
Isabel-50M 54M 38.74
Supra-50M-Base 52M 38.45
Archaea-74M-V1.1 74M 37.96

It leads the sub-100M class on ARC-easy and PIQA, and its ArithMark-2 score of 52.48 is the second highest in that class.

Training

Trained from scratch with a custom digit-aware 32k tokenizer, then refined on a curated mix of educational, science, commonsense and reading-comprehension data, plus a large synthetic integer arithmetic set covering addition, subtraction, multiplication, exact division, mixed multi-operator expressions and parenthesised expressions. No benchmark test items were used at any stage.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-50M")
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-50M")

prompt = "16 + 4 * 3 ="
print(tok.decode(model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=6)[0]))

Example outputs

Real, unedited outputs from this checkpoint.

Prompt Output
Question: What force pulls objects toward the Earth? gravity.
Question: What gas do humans need to breathe to survive? oxygen.
Question: What do we call the process by which plants make food? photosynthesis.

License

Apache-2.0.

Built by Ideoa Labs.

Downloads last month
686
Safetensors
Model size
54.1M params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Space using MaliosDark/Nexus-Erebus-50M 1