SupraLabs/SupraCMA-8M
Text Generation • 9.08M • Updated • 373 • 9
🏗️ Building on HF
Mr Munk
GODELEV
AI & ML interests
High schooler by day, LLM builder by night. Driven by a deep love for both Physics and AI. Currently spending my runtime building on Hugging Face, experimenting with transformer architectures, and training custom LLMs.
Recent Activity
liked a model 2 days ago
SupraLabs/SupraCMA-8M reacted to Banaxi-Tech's post with 👍 2 days ago
We're excited to release BananaMind 2 Mini the first model in our BananaMind 2 series!
BananaMind 2 Mini features a custom digit-aware BPE tokenizer that keeps every digit isolated, fixing the core arithmetic weakness of our previous models. It's trained on 30B tokens from FineWeb-Edu, DCLM, Cosmopedia-v2 and FineMath-4+, and already outperforms Pythia-31M despite having fewer parameters.
Check it out at
https://huggingface.co/BananaMind/BananaMind-2-Mini
BananaMind 2 Pro is coming next. Apache 2.0. updated a dataset 4 days ago
GODELEV/SomethingNewOrganizations
None yet
liked a
model 2 days ago
reacted to Banaxi-Tech's post with 👍 2 days ago
Post
253
We're excited to release BananaMind 2 Mini the first model in our BananaMind 2 series!
BananaMind 2 Mini features a custom digit-aware BPE tokenizer that keeps every digit isolated, fixing the core arithmetic weakness of our previous models. It's trained on 30B tokens from FineWeb-Edu, DCLM, Cosmopedia-v2 and FineMath-4+, and already outperforms Pythia-31M despite having fewer parameters.
Check it out at
BananaMind/BananaMind-2-Mini
BananaMind 2 Pro is coming next. Apache 2.0.
BananaMind 2 Mini features a custom digit-aware BPE tokenizer that keeps every digit isolated, fixing the core arithmetic weakness of our previous models. It's trained on 30B tokens from FineWeb-Edu, DCLM, Cosmopedia-v2 and FineMath-4+, and already outperforms Pythia-31M despite having fewer parameters.
Check it out at
BananaMind/BananaMind-2-Mini
BananaMind 2 Pro is coming next. Apache 2.0.
- 1 reply
updated a
dataset 4 days ago
published a
dataset 4 days ago
liked a
model 5 days ago
updated a
dataset 6 days ago
liked a
model 6 days ago
liked a
dataset 7 days ago
liked a
model 8 days ago
liked 4
datasets 9 days ago
liked 2
models 10 days ago
reacted to Banaxi-Tech's post with 👀 11 days ago
Post
4231
Today we are releasing BananaMind-KV1-8M-2Bit-Experimental, a KV-cache-aware trained model that stores its generation KV cache in 2-bit precision instead of the usual 16-bit precision.
Result: 5.33x smaller KV cache vs FP16, with 0.0916 mean KLD against a 16-bit KV cache reference on WikiText-2.
Model: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
The important part: this is not just post-training KV cache quantization.
Instead we take the BitNet approach.
KV1 is trained with a 2-bit-aware K/V path. Instead of training a normal model and quantizing the cache afterwards, the model learns during training to operate under the low-bit KV constraint, closer in spirit to the BitNet idea of training for the low-bit regime.
During generation, each K/V vector is quantized into 4 affine levels and packed into uint8 tensors, with four 2-bit values stored per byte.
WikiText-2 eval vs 16-bit KV cache reference:
Mean KLD: 0.0916 nats/token
Mean KLD: 0.1322 bits/token
Average KV cache shrink vs FP16: 5.33x
Evaluated positions: 372,675
If this actually gets used in models like Qwen or Gemma, then it may be possible to run 128K or even 256K Context on a Normal Machine!
Try it here: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
Code: https://github.com/Banaxi-Tech/kv1
Result: 5.33x smaller KV cache vs FP16, with 0.0916 mean KLD against a 16-bit KV cache reference on WikiText-2.
Model: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
The important part: this is not just post-training KV cache quantization.
Instead we take the BitNet approach.
KV1 is trained with a 2-bit-aware K/V path. Instead of training a normal model and quantizing the cache afterwards, the model learns during training to operate under the low-bit KV constraint, closer in spirit to the BitNet idea of training for the low-bit regime.
During generation, each K/V vector is quantized into 4 affine levels and packed into uint8 tensors, with four 2-bit values stored per byte.
WikiText-2 eval vs 16-bit KV cache reference:
Mean KLD: 0.0916 nats/token
Mean KLD: 0.1322 bits/token
Average KV cache shrink vs FP16: 5.33x
Evaluated positions: 372,675
If this actually gets used in models like Qwen or Gemma, then it may be possible to run 128K or even 256K Context on a Normal Machine!
Try it here: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
Code: https://github.com/Banaxi-Tech/kv1
- 4 replies
reacted to Quazim0t0's post with 👀 11 days ago
Post
2396
Created research language model whose channel-mixing block is not an MLP. It is a differentiable Neighbour-Sensing fungal-colony-growth model: each token is expanded into a colony of hyphal tips that grow in a bounded latent region, sense a shared density field, and steer their own growth — the "MLP" is replaced by a few differentiable steps of colony growth, read back out into the hidden state.
Quazim0t0/Mycel-LM-79M
Also the original SpikeWhale project — the one that sparked all the other SpikeWhale related projects. Every spiking primitive here is hand-written in plain PyTorch: the leaky integrate-and-fire (LIF) neuron dynamics, the fast-sigmoid surrogate gradient, and the backprop-through-time training loop. No snntorch, no spikingjelly, no norse, no bindsnet — the network is a genuine from-scratch SNN.
Quazim0t0/SpikeWhale-SNN-216M
Quazim0t0/Mycel-LM-79M
Also the original SpikeWhale project — the one that sparked all the other SpikeWhale related projects. Every spiking primitive here is hand-written in plain PyTorch: the leaky integrate-and-fire (LIF) neuron dynamics, the fast-sigmoid surrogate gradient, and the backprop-through-time training loop. No snntorch, no spikingjelly, no norse, no bindsnet — the network is a genuine from-scratch SNN.
Quazim0t0/SpikeWhale-SNN-216M
- 5 replies
liked a
Space 12 days ago
[bot] Conversion to Parquet
#1 opened 15 days ago
by 
parquet-converter