This model is broken.
This model hallucinates badly, even if this is the correct weights, then meta couldn't pull this off correctly. this has zero value except being a spectacle.
I agree! This model does have some bad hallucinations... but it also has the same hallucinations on the API playground (and anyone else with access can confirm that). I also agree that this model is useless as anything other than an artifact of the times.
My main hilarious example of this I found when I was testing is that it somehow does not know what horseshoe theory is, both on the API and real weights
it's possible that meta continue trained 3.3 version the same amount of tokens for all models sizes, 8b didn't make it because it got saturated and they held it back.
this is not totally broken, it's just...well...lobotomized
What's weird is it's honestly not lobotomized, its actually decently smart! It just has the world knowledge of a squirrel
idk, it was making very dumb mistakes on my end, I don't think you need world knowledge to write a decent roast yet it failed even that.
that's just llama 3 for you tbh