The app crashes for some underrepresented languages
E.g. when I enter "Erzya", I see a blank screen.
Seems to be a bug.
Indeed, that shouldn't be, thanks for reporting! – @jonas can you look into this later this week? :)
Hi
@cointegrated
! Should be fixed now.
The reason the app was crashing (simplifying) is that no AI model was evaluated yet on Erzya. In a way this is also what we strive to showcase with our tool: the existing gaps and coverage for low resource languages. In this case, however, the reason is that we have only ran the evaluations only for the top 100 spoken languages yet. In future releases they will all be included.
I made this more clear in the frontend when searching for such cases!
Thanks for spotting!
@boffire
Thanks for reporting!
Kabyle is now included in the evaluation 🥳
Note the general limitation that we have currently only evaluated 10 samples for language x model x task, so individual results in a language-specific leaderboard are not very robust.