👋 Coding Training Code

Banaxi PRO

Banaxi-Tech

26 21

Banaxi-Tech

AI & ML interests

SLMs, training from scratch, LoRA, TTS, Ternary models

Recent Activity

reacted to theirpost with 👍 about 22 hours ago

We're excited to release BananaMind 2 Mini the first model in our BananaMind 2 series! BananaMind 2 Mini features a custom digit-aware BPE tokenizer that keeps every digit isolated, fixing the core arithmetic weakness of our previous models. It's trained on 30B tokens from FineWeb-Edu, DCLM, Cosmopedia-v2 and FineMath-4+, and already outperforms Pythia-31M despite having fewer parameters. Check it out at https://huggingface.co/BananaMind/BananaMind-2-Mini BananaMind 2 Pro is coming next. Apache 2.0.

liked a model 1 day ago

BananaMind/BananaMind-2-Mini

new activity 5 days ago

AxiomicLabs/Open_SLM_Leaderboard:Add https://huggingface.co/BananaMind/BananaMind-2-Mini

View all activity

Organizations

reacted to their post with 👍 about 22 hours ago

Post

166

We're excited to release BananaMind 2 Mini the first model in our BananaMind 2 series!

BananaMind 2 Mini features a custom digit-aware BPE tokenizer that keeps every digit isolated, fixing the core arithmetic weakness of our previous models. It's trained on 30B tokens from FineWeb-Edu, DCLM, Cosmopedia-v2 and FineMath-4+, and already outperforms Pythia-31M despite having fewer parameters.
Check it out at
BananaMind/BananaMind-2-Mini

BananaMind 2 Pro is coming next. Apache 2.0.

1 reply

posted an update 5 days ago

Post

166

We're excited to release BananaMind 2 Mini the first model in our BananaMind 2 series!

BananaMind 2 Mini features a custom digit-aware BPE tokenizer that keeps every digit isolated, fixing the core arithmetic weakness of our previous models. It's trained on 30B tokens from FineWeb-Edu, DCLM, Cosmopedia-v2 and FineMath-4+, and already outperforms Pythia-31M despite having fewer parameters.
Check it out at
BananaMind/BananaMind-2-Mini

BananaMind 2 Pro is coming next. Apache 2.0.

1 reply

posted an update 8 days ago

Post

128

BananaMind 2 Mini is now officially training!
Live Dashboard: https://huggingface.co/BananaMind/BananaMind-2-Mini-Status (updates every few minutes)
We are training it on 30B Tokens and a mix of DCLM, FineWeb Edu, FineMath, and Cosmopedia!
We estimate the training to take around 2 days.
Follow us to know when it releases!

replied to their post 9 days ago

Will be hard to measure difference though because these models are small still.

replied to their post 9 days ago

Great idea, I will try soon! But i am currently focusing on training my new Models.

replied to their post 9 days ago

UPDATE:
Token mix for Pro Is Confirmed!
FineWeb-Edu: 49.79B
DCLM: 33.91B
FineMath: 4.5B
NPSet-Python: 2.51B
Cosmopedia V2 10.00B

replied to their post 9 days ago

This comment has been hidden

posted an update 10 days ago

Post

4227

Today we are releasing BananaMind-KV1-8M-2Bit-Experimental, a KV-cache-aware trained model that stores its generation KV cache in 2-bit precision instead of the usual 16-bit precision.

Result: 5.33x smaller KV cache vs FP16, with 0.0916 mean KLD against a 16-bit KV cache reference on WikiText-2.

Model: BananaMind/BananaMind-KV1-8M-2Bit-Experimental

The important part: this is not just post-training KV cache quantization.
Instead we take the BitNet approach.

KV1 is trained with a 2-bit-aware K/V path. Instead of training a normal model and quantizing the cache afterwards, the model learns during training to operate under the low-bit KV constraint, closer in spirit to the BitNet idea of training for the low-bit regime.

During generation, each K/V vector is quantized into 4 affine levels and packed into uint8 tensors, with four 2-bit values stored per byte.

WikiText-2 eval vs 16-bit KV cache reference:

Mean KLD: 0.0916 nats/token
Mean KLD: 0.1322 bits/token
Average KV cache shrink vs FP16: 5.33x
Evaluated positions: 372,675

If this actually gets used in models like Qwen or Gemma, then it may be possible to run 128K or even 256K Context on a Normal Machine!
Try it here: BananaMind/BananaMind-KV1-8M-2Bit-Experimental

Code: https://github.com/Banaxi-Tech/kv1

4 replies

replied to their post 11 days ago

classic autoregressive architecture

replied to their post 11 days ago

Thanks for the post, now my primary question: will those models use digit level tokenizer and how do you handle instruction following? I'm happy to see small models beating older bigger models.

Yes we will use a digit level tokenizer, and Instruction ones are coming later but our Primary focus is the base first.

replied to their post 11 days ago

sota?? i tried creating one on a rtx pro 6000 blackwell server edition gpu, and it was worse than any 90m~ model. are you using your datasets or open source one?

We are going to be using FineWeb Edu, FineMath and some more, and yes all Open Source Token count we dont know yet because we may change but we think ~90B

replied to their post 12 days ago

oh! i saw this for first time. what's it? a model like NanoBanana? what's it goal?

LLM

posted an update 12 days ago

Post

10730

A new model is coming!
Its going to take a long time on my 5070 Ti so expect a release in ~1 month.
We think this model is going to be SOTA For its size.
Our Mini Version will be 25M Parameters and Pro with 140M.
The Pro version has a 3072 Context Window (Extensible to up to 6K with RoPE) And the Mini version has a context window of 4096 (Up to 8K with RoPE)
Meanwhile we are currently working on a Instruct Version of our BananaMind 1.5 Base.

The training will start this weekend

We are very exited to release it when its done!

12 replies

posted an update 14 days ago

Post

391

📱 TinyPhoneLM - LLMs on a Phone
I built TinyPhoneLM because I wanted to see how far tiny local LMs can go on a real Android phone.
Not just a server app.
Not just an API wrapper.
Not “AI on your phone” that secretly sends everything somewhere else.

TinyPhoneLM allows you to run small language models directly on android. It uses llama.cpp via JNI. We have alot of options for default models + custom GGUF Import Supported. I am running Qwen3.5 4B Locally on my Redmi Note 12 Pro 5G at 4 tokens per second, that may seem slow but that it even runs on my phone is insane. I can also run Qwen3.5 0.8B at 10TPS!
Look at this Chart From Artificial Analysis.
Qwen3.5 4B is Better than GPT 4.1 and GPT 5 Mini at minimal reasoning!
And even the smallest 800M Parameter Qwen3.5 0.8B still beats GPT 3.5 Turbo!

The bad news: To get it on the play store we need 12 Testers

Please only submit your Google Play email if you have a Android phone
If you want to test TinyPhoneLM, enter your Google Play email here:

👉 https://docs.google.com/forms/d/1LqkT2pUHbalSUV50M8PX8m7M6S122ip0cWcbKcytcXk/viewform
I would really appreciate the help if you get a tester!