scaling laws
10 billion tokens on a sub 1m param model? llms aren't that overparameterised lmao, plus you have a 5090, you can easily go up to 100M params for pretrain so why don't you?
I can't speak for them, but it is probably more about efficiency and rapid prototyping.
10 billion tokens on a sub 1m param model? llms aren't that overparameterised lmao, plus you have a 5090, you can easily go up to 100M params for pretrain so why don't you?
I like experimenting with small models, and this is a quick way to do that.
The largest model is 50M :P
The goal is to see how good a super small model can get
i completely support the premise of this but i would be very surprised if it could even form sentences
This model kinda already does
10 billion tokens on a sub 1m param model? llms aren't that overparameterised lmao, plus you have a 5090, you can easily go up to 100M params for pretrain so why don't you?
that's what im trying to tell him, but, he really don't want
shut up the whole point of this is 1m param intelligence on small models are the future dork
shut up the whole point of this is 1m param intelligence on small models are the future dork
Finally someone agrees
Scaling laws have changed overtime, people used to go by chinchilla scaling which is literally minuscule compared to nowadays
Scaling laws have changed overtime, people used to go by chinchilla scaling which is literally minuscule compared to nowadays
nah lil bro i need gpt 5.6 sol on 9 params frick scaling laws
There exists a information theoretic maximum number of information you can put into the model.
That number is enormous.
But using all of it does not mean the model will be superhuman, instead it will likely:
- hallucinate
- write gibberish
- repeat or loop
- not be able to reduce loss further, meaning it won't learn and saturate.
There exists a information theoretic maximum number of information you can put into the model.
That number is enormous.
But using all of it does not mean the model will be superhuman, instead it will likely:
- hallucinate
- write gibberish
- repeat or loop
- not be able to reduce loss further, meaning it won't learn and saturate.
Not how scaling works.
There exists a information theoretic maximum number of information you can put into the model.
That number is enormous.
But using all of it does not mean the model will be superhuman, instead it will likely:
- hallucinate
- write gibberish
- repeat or loop
- not be able to reduce loss further, meaning it won't learn and saturate.
Not how scaling works.
Real in the downstream sense but hes correct about compressibility of language
Any piece of research seems useless until the exact moment you actually need it. Beyond what this specific model can or can't do right now, the data and insights gained from pushing it this far are valuable. Right now, people are out here doing whatever they want. They are using sloths to fine-tune, breeding llamas, and ordering Chinese takeout models to go.
If it feels like this project doesn't matter, it's only because people are looking at it solely through the lens of 'what can this model do for me right now? and since it can't do what I want it to, therefore it's a waste of compute.' It's wild to spawn a judgment call on what this compute went towards when half the world is wasting hardware playing video games 8 hours a day. I don't see the need to judge what another person does with their own hardware lol.
shut up the whole point of this is 1m param intelligence on small models are the future dork
gng ✌️🥀
shut up the whole point of this is 1m param intelligence on small models are the future dork
gng ✌️🥀
huh?
he is right though.
shut up the whole point of this is 1m param intelligence on small models are the future dork
gng ✌️🥀
huh?
he is right though.
this he said "1M param intelligence" is not real yet, that's what im trying to say, but, i really believe in you @CompactAI , it would be completely crazy if a 1M param model talk like a 50M, but, thats just not real yet.
What is going on here, jesus
What is going on here, jesus
idk lol
shut up the whole point of this is 1m param intelligence on small models are the future dork
gng ✌️🥀
shush laxionrp labs
shut up the whole point of this is 1m param intelligence on small models are the future dork
gng ✌️🥀
huh?
he is right though.this he said "1M param intelligence" is not real yet, that's what im trying to say, but, i really believe in you @CompactAI , it would be completely crazy if a 1M param model talk like a 50M, but, thats just not real yet.
its real bro go back to ur lab and train 9999t model no one care
shut up the whole point of this is 1m param intelligence on small models are the future dork
gng ✌️🥀
shush laxionrp labs
shush larpxpy
its real bro go back to ur lab and train 9999t model no one care
you're really trying to say that to defend Glint Research, but then you see they are training a 6T params model lol
this he said "1M param intelligence" is not real yet, that's what im trying to say, but, i really believe in you @CompactAI , it would be completely crazy if a 1M param model talk like a 50M, but, thats just not real yet.
HOWEVER "1M param intelligence" just means it has intelligence, which is subjective, so who cares, this entire convo is pointless, imo it has intelligence because its subjective.
to be fair, that is a experiment model. Small models are our #1 priority.
also, the 6T model is usable at 15tps on a basic consumer CPU
Pls bro make a total revolution with Aureole 😭😭😭😭😭😭
shut up the whole point of this is 1m param intelligence on small models are the future dork
gng ✌️🥀
shush laxionrp labs
shush larpxpy
it would be larpxy but ok
to be fair, that is a experiment model. Small models are our #1 priority.
also, the 6T model is usable at 15tps on a basic consumer CPU
idk wy axionlab is complaining they also do small models we need gpt 900 universe doing 500m tps 2 params