appvoid

Following

Hey

following

mine are all <300M

Great, following already!

Surely I do!

following

is there anything you have always wanted to achieve? that idea you know you want to be true? is possible if any

keep working 💪

achieve maximum benchmark score per param

lol

you're talking about robust performance and not benchmaxxing, right? right?

is there anything you have always wanted to achieve? that idea you know you want to be true? is possible if any

keep working 💪

following you!

ok

I want to follow people that also make super cool small models so comment here to follow you

cool, mind sharing it?

I'm in the works for one actually. I have been talking about it forever but I want to make sure is at the level of people's expectation.

I have been always fascinated to discover what's the minimum core of actual usefulness.

5 years ago I thought it was 1 billion parameters, but recently I discovered a 350m that is better than most 1b models.

350m is not perfect but as of today, if you go lower you lose general language understanding.

I think we agree more than it seems. I am not saying manual review should replace diversity metrics.

Skeleton counts and entropy are useful, but they only catch what the parser measures. A batch can have many distinct skeletons and still repeat the same reasoning, tone, difficulty, or task patterns.

My point is to catch that early, while the batch is still small enough to change the prompts or generation strategy. After 100k rows, deduplication cannot recover the missing diversity.

The safest rail is both: batch-level diversity metrics and periodic human review.

I want to clarify that obviously still we need to generate with the best models we have but evaluate the batches manually be it 100 or 1000. Even if it's a quick read over examples. Never EVER let an AI generate without rails. I was too confident on frontier models outputs just to get dissapointed later.

I think is the second thing. A small model that enters a strange context may produce a relatively flat distribution. It does not have a strong idea of what should come next but still it has to pick something up. Anti-repetition behavior from data consumes capacity, the less capacity, the more likely your model will suffer from this bias.

That's a simplified view I have on self-reinforcing degeneration mechanism which you can read about here btw: https://arxiv.org/abs/2109.08705