Wop

I suppose you can see for yourself

Right now, a finished run is written to leaderboard.json (model row) but also models.json (a file that is ready to upload to repo & make pr) for 'lazzier people'- but best approach and less likely to be invalid.

I quite no longer wish to have to deal with this problem..

I would greatly appreciate a PR that fixes it than a discussion from which I won't understand too much

I am mainly the most active person in benchlabs, I cannot multitask but neither continue on this, since you are here, I greatly appreciate all your dedication to improving this reproductibility system

However, even as I find this difficult, I don't want to let quality away, but for now, I am going to take a break from the leaderboard and datasets.. I will be happy to accept PR merges when they arrive.

I don't know what my focus will be redirected to.
But ive seen a lot of interest for PixelModel (a very cheap yet smart text to image model) and I think making upgrades there is my next move.

I will continue to reply here (and everywhere else), but I won't contribute myself to the leaderboard for a while

Thanks again

It seems very important

Another one tho:

One thing that is not yet saved is the exact command arguments used for the run, like Max-tokens , which when modified, it can return different results.

None of them is cheap

Another thing, I like autonomous but also getting feedback or contribution of thinking for things like 'what do you think', 'what do you propose'

Take it then..
I'll let you implement whatever upgrades in reproducibility
One question I haven't been able to get (as a second language english user)
"Of those inputs, which ones does BenchLabs not pin at all right now?", can you pharapharse this one to be more specific? Thank you