FromZero

We build stuff and share it with the community.

That's pretty much it. And yes, "stuff" was the right word to use.

Why "FromZero"?

FromZero is inspired by Re:Zero - Starting Life in Another World. Just like the main character, all of our models begin from zero.

Thankfully, they don't spend their time dying repeatedly or having mental breakdowns.

I mean, training is already painful enough.

Our goal

Well, there isn't actually a definitive goal we're chasing. I mean, yeah, we want to be the greatest. But so does everyone else.

We're just here to build weird models, share them with the community, and grow side-by-side with everyone.

Current Models

• Zero-v0.1-150M: Our highest-performing and most recent model, boasting a total of 151.6M parameters and a 2048-token context window.
• Er-Large: A 31M-parameter model trained on 34B tokens, achieving near state-of-the-art performance across multiple benchmarks.
• MrPong: Our SOTA Ping Pong reinforcement learning (RL) agent, trained for over 10M steps against nine different opponents.

Future Releases

• Zero-v1.0-144M: The next-generation model in the Zero family. It will be trained on 133B tokens and will introduce a new custom architecture featuring Engram Conditional Memory, mHC, and XSA GQA Attention.
• JetonCount-2: The second generation of JetonCount. It will be trained on 25 different datasets and 50 tokenizers, featuring an improved architecture.
• Max-v1.0-3M: A 3M-parameter model that will be trained on 7B tokens. It will use the same architecture as Zero-v1.0, with Hadamard FFNs and SwiGLU intervals.
• Negative-Engram-150M-A67k: An experimental model inspired by Negative, featuring a significantly larger Engram table.

Partners

If you like us, then you might also like:

AxiomicLabs | BenchLabs | QyrouLabs | GODELEV

Discord

Join our Discord community: FromZero

YouTube

Visit our YouTube channel at https://www.youtube.com/@fromziro-ia.

If You Want to Join Us

We are always looking for more members. If you'd like to join, open a discussion under this README and tell us why. We'll review your request and either approve or reject it.