Post
58
Mini-Beatrix-2s is cooking with full splat attention through and through. This model is still trigram, I did this to get a baseline because there's already a trigram model to compare to. This one should be done in a few days and ought to be substantially more intelligent than the first.
Specs are;
Around 220m params, 4096 context window, d1024 model size, splat 128, 1024, 1024, 1024, and so on, 3 experts per block, information banks for storage and retrieval, and a lot of technical knowhow between A to B.
Differences with V2;
Special tokens are implemented byte-directly, so the model will have no problem recognizing an array of special tokens such as DOC, EOF, and a multitude of others.
Suffice it to say, this model is bigger than the first at about 2x. Not just bigger though, estimated to be roughly 8x more intelligent based on the measures.
That being said, the actual model needs to be substantially larger to encompass the full space. The measured space is considerably larger through the small tests for stability, however the full 900m version runs at only around 8k tokens per second with an anchor count of 131,000 and a matching number of heads. This means the full train would require roughly 26 days on a rtx 6000 pro blackwell, which is substantially beyond the expectation curve.
So the smaller one will do for now until I can secure a bit of funding. In any case, the tokenizer system will be implemented on this version after a stable run completes.
Specs are;
Around 220m params, 4096 context window, d1024 model size, splat 128, 1024, 1024, 1024, and so on, 3 experts per block, information banks for storage and retrieval, and a lot of technical knowhow between A to B.
Differences with V2;
Special tokens are implemented byte-directly, so the model will have no problem recognizing an array of special tokens such as DOC, EOF, and a multitude of others.
Suffice it to say, this model is bigger than the first at about 2x. Not just bigger though, estimated to be roughly 8x more intelligent based on the measures.
That being said, the actual model needs to be substantially larger to encompass the full space. The measured space is considerably larger through the small tests for stability, however the full 900m version runs at only around 8k tokens per second with an anchor count of 131,000 and a matching number of heads. This means the full train would require roughly 26 days on a rtx 6000 pro blackwell, which is substantially beyond the expectation curve.
So the smaller one will do for now until I can secure a bit of funding. In any case, the tokenizer system will be implemented on this version after a stable run completes.
