Ash

Thank you for that link, i might train future models with mamba3 right now though im going to stick with mamba2, mamba2 vs mamba3 will probably be one of the things i test in the hunt for the optimal mamba models.

I also basically live under a rock, i somehow miss a lot of things in the AI space..

tbh i didnt actually know that mamba3 was out... but thanks for letting me know.

Thank you ContextReq!

Unfortunately as far as i am am able to get the models to run, i havnt been able to get CPU working, when i do though i will let everyone know..

@CodeSoft If you want some tips for training decent models in this size range what i did/do is at least have fineweb-edu and cosmopedia-v2, and it would be recommended to have DCLM, math data like FineMath-4+, and a tiny bit of code data like python but for 5M you would probably be fine skipping code, use the Muon opimizer for 2D weights and AdamW for everything else for 5M and i would aim for at least 500M tokens, and i would settle for a 2048 vocab tokenizer just to leave the most room possible for non embedding params. I wish you the best of luck and i hope this helps

you know, i'll actually test all of this. i can run some experiments comparing:

Architecture Mamba : Attention
Mamba-only ∞:1
Mostly Mamba 7:1
Current model 3:1
More attention 1:1
Transformer-only 0:1

Then we can actually see which ratio performs best at this scale.

As far as I know, I don't think the attention layers were dominating the gradients, but I'll measure that when I run these experiments.

When I get to them, the models will probably be released under https://huggingface.co/basically-experimental