evaleval/every_eval_ever
Updated
โข
35
AI & ML interests
Weโre building a research coalition on evaluating evaluations (EvalEval)! Hosted by Hugging Face, University of Edinburgh, and EleutherAI.
Recent Activity
View all activity
evijitย
published
a
dataset
13 days ago
Post
2491
AI for Scientific Discovery Won't Work Without Fixing How We Collaborate.
My co-author @cgeorgiaw and I just published a paper challenging a core assumption: that the main barriers to AI in science are technical. They're not. They're social.
Key findings:
๐จ The "AI Scientist" myth delays progress: Waiting for AGI devalues human expertise and obscures science's real purpose: cultivating understanding, not just outputs.
๐ Wrong incentives: Datasets have 100x longer impact than models, yet data curation is undervalued.
โ ๏ธ Broken collaboration: Domain scientists want understanding. ML researchers optimize performance. Without shared language, projects fail.
๐ Fragmentation costs years: Harmonizing just 9 cancer files took 329 hours.
Why this matters: Upstream bottlenecks like efficient PDE solvers could accelerate discovery across multiple sciences. CASP mobilized a community around protein structure, enabling AlphaFold. We need this for dozens of challenges.
Thus, we're launching Hugging Science! A global community addressing these barriers through collaborative challenges, open toolkits, education, and community-owned infrastructure. Please find all the links below!
Paper: AI for Scientific Discovery is a Social Problem (2509.06580)
Join:
hugging-science
Discord: https://discord.com/invite/VYkdEVjJ5J
My co-author @cgeorgiaw and I just published a paper challenging a core assumption: that the main barriers to AI in science are technical. They're not. They're social.
Key findings:
๐จ The "AI Scientist" myth delays progress: Waiting for AGI devalues human expertise and obscures science's real purpose: cultivating understanding, not just outputs.
๐ Wrong incentives: Datasets have 100x longer impact than models, yet data curation is undervalued.
โ ๏ธ Broken collaboration: Domain scientists want understanding. ML researchers optimize performance. Without shared language, projects fail.
๐ Fragmentation costs years: Harmonizing just 9 cancer files took 329 hours.
Why this matters: Upstream bottlenecks like efficient PDE solvers could accelerate discovery across multiple sciences. CASP mobilized a community around protein structure, enabling AlphaFold. We need this for dozens of challenges.
Thus, we're launching Hugging Science! A global community addressing these barriers through collaborative challenges, open toolkits, education, and community-owned infrastructure. Please find all the links below!
Paper: AI for Scientific Discovery is a Social Problem (2509.06580)
Join:
hugging-science
Discord: https://discord.com/invite/VYkdEVjJ5J
felfriย
authored
a
paper
28 days ago
Post
693
the french ministry of culture releases their first conversation datasets on huggingface ๐๐ป
ministere-culture/comparia-conversations
ministere-culture/comparia-conversations
Post
2853
๐ค As AI-generated content is shared in movies/TV/across the web, there's one simple low-hanging fruit ๐ to help know what's real: Visible watermarks. With the Gradio team, I've made sure it's trivially easy to add this disclosure to images, video, chatbot text. See how: https://huggingface.co/blog/watermarking-with-gradio
Thanks to the code collab in particular from @abidlabs and Yuvraj Sharma.
Thanks to the code collab in particular from @abidlabs and Yuvraj Sharma.
Post
669
COMPUTER CONTROL IS ON-DEVICE !
๐ก๐ค 78 % of EU smart-home owners DONโT trust cloud voice assistants.
So we killed the cloud.
Meet Extรฉ: a palm-sized Android device that sees, hears & speaks your language - 100 % offline, 0 % data sent anywhere.
๐ We submitted our technologies for consideration to the Liquid AI hackathon.
๐ Dataset: 79 k UI-action pairs on Hugging Face (largest Android-control corpus ever) Tonic/android-operator-episodes
โก Model: 98 % task accuracy, 678MB compressed , fits on existing android devices ! Tonic/l-android-control
๐ค๏ธ Experiment Tracker : check out the training on our TrackioApp Tonic/l-android-control
๐ฎ Live Model Demo: Upload an Android Screenshot and instructions to see the model in action ! Tonic/l-operator-demo
Built in a garage, funded by pre-orders, no VC. Now weโre scaling to 1 k installer units.
Weโre giving 50 limited-edition prototypes to investors , installers & researchers who want to co-design the sovereign smart home.
๐ Drop โEUSKERAโ in the comments if you want an invite, tag a friend who still thinks Alexa is โconvenient,โ and smash โฅ๏ธ if AI should belong to people - not servers.
๐ก๐ค 78 % of EU smart-home owners DONโT trust cloud voice assistants.
So we killed the cloud.
Meet Extรฉ: a palm-sized Android device that sees, hears & speaks your language - 100 % offline, 0 % data sent anywhere.
๐ We submitted our technologies for consideration to the Liquid AI hackathon.
๐ Dataset: 79 k UI-action pairs on Hugging Face (largest Android-control corpus ever) Tonic/android-operator-episodes
โก Model: 98 % task accuracy, 678MB compressed , fits on existing android devices ! Tonic/l-android-control
๐ค๏ธ Experiment Tracker : check out the training on our TrackioApp Tonic/l-android-control
๐ฎ Live Model Demo: Upload an Android Screenshot and instructions to see the model in action ! Tonic/l-operator-demo
Built in a garage, funded by pre-orders, no VC. Now weโre scaling to 1 k installer units.
Weโre giving 50 limited-edition prototypes to investors , installers & researchers who want to co-design the sovereign smart home.
๐ Drop โEUSKERAโ in the comments if you want an invite, tag a friend who still thinks Alexa is โconvenient,โ and smash โฅ๏ธ if AI should belong to people - not servers.
Post
2382
Tremendous quality of life upgrade on the Hugging Face Hub - we now have auto-complete emojis ๐ค ๐ฅณ ๐ ๐ ๐
Get ready for lots more very serious analysis on a whole range of topics from yours truly now that we have unlocked this full range of expression ๐ ๐ค ๐ฃ ๐
Get ready for lots more very serious analysis on a whole range of topics from yours truly now that we have unlocked this full range of expression ๐ ๐ค ๐ฃ ๐
Post
670
๐๐ปโโ๏ธ Hey there folks ,
Just wanted to annouce ๐ญSmolFactory : it's the quickest and best way to finetune SmolLM3 and GPT-OSS-20B on huggingface !
Basicaly it's an app you can run on huggingface by duplicating the space and running your training directly on huggingface GPUs .
It will help you basically select datasets and models, fine tune your model , make an experiment tracker you can use on your mobile phone , push all your model card and even automatically make a demo for you on huggingface so you can directly test it out when it's done !
check out the blog to learn more : https://huggingface.co/blog/Tonic/smolfactory
or just try the app directly :
Tonic/SmolFactory
you can vibe check the cool models I made :
French SmolLM3 : Tonic/Petite-LLM-3
Medical GPT-OSS : Tonic/med-gpt-oss-20b-demo
check out the model cards :
multilingual reasoner (gpt-oss) - Tonic/gpt-oss-20b-multilingual-reasoner
med-gpt-oss : Tonic/med-gpt-oss-20b
petite-elle-l-aime : Tonic/petite-elle-L-aime-3-sft
github repo if you like command line more than gradio : https://github.com/josephrp/smolfactory
drop some likes on these links it's really much appreciated !
feedback and PRs are welcome !
Just wanted to annouce ๐ญSmolFactory : it's the quickest and best way to finetune SmolLM3 and GPT-OSS-20B on huggingface !
Basicaly it's an app you can run on huggingface by duplicating the space and running your training directly on huggingface GPUs .
It will help you basically select datasets and models, fine tune your model , make an experiment tracker you can use on your mobile phone , push all your model card and even automatically make a demo for you on huggingface so you can directly test it out when it's done !
check out the blog to learn more : https://huggingface.co/blog/Tonic/smolfactory
or just try the app directly :
Tonic/SmolFactory
you can vibe check the cool models I made :
French SmolLM3 : Tonic/Petite-LLM-3
Medical GPT-OSS : Tonic/med-gpt-oss-20b-demo
check out the model cards :
multilingual reasoner (gpt-oss) - Tonic/gpt-oss-20b-multilingual-reasoner
med-gpt-oss : Tonic/med-gpt-oss-20b
petite-elle-l-aime : Tonic/petite-elle-L-aime-3-sft
github repo if you like command line more than gradio : https://github.com/josephrp/smolfactory
drop some likes on these links it's really much appreciated !
feedback and PRs are welcome !
iamgroot42ย
authored
a
paper
2 months ago
Post
2934
New work from my socially-minded colleagues at Hugging Face, creating some foundations for AI companionship behavior evaluation.
Evaluation Dataset: AI-companionship/INTIMA
Paper: AI-companionship/INTIMA
Work from @giadap , @frimelle , @yjernite .
Evaluation Dataset: AI-companionship/INTIMA
Paper: AI-companionship/INTIMA
Work from @giadap , @frimelle , @yjernite .
- 2 replies
yjerniteย
authored
3
papers
3 months ago
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Paper
โข
2406.16746
โข
Published
In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI
Paper
โข
2503.16861
โข
Published
โข
1
A Different Approach to AI Safety: Proceedings from the Columbia Convening on Openness in Artificial Intelligence and AI Safety
Paper
โข
2506.22183
โข
Published
โข
1
Post
437
๐ค ICYMI: Yesterday, Hugging Face and OpenAI partnered to bring open source GPT to the public. This is a Big Deal in "AI world".
0. Common ground setting: OpenAI is the ChatGPT people. An โopen sourceโ model is one whose weights are available โ that means the model can be โyoursโ.
1. You donโt have to interact with the company directly, nor give them your interactions, to use the system. The company can't "surveil" you.
2. You can evaluate the unique contributions of their SOTA model much more rigorously than you can when there are collections of models+code behind a closed API. You can find out specifically what the model can and can't do.
3. And you can directly customize it for whatever you'd like. Fine-tuning, wherein you give the model data that's tailored to your use cases and train it some more on that data, is trivial* when you have the model weights.
*Provided you have the compute.
4. You can directly benchmark whatever you'd like. Biases? Energy usage? Strengths/weaknesses? Go for it. You wants it you gots it--this transparency helps people understand SOTA *in general*, not just for this model, but points to, e.g., what's going on with closed Google models as well.
5. One of the most powerful things about "openness" that I've learned is that it cultivates ecosystems of collaborators building on top of one another's brilliance to make systems that are significantly better than they would be if created in isolation.
But, caveat wrt my own philosophy...
6. I do not take it as a given that advancing LLMs is good, and have a lot more to say wrt where I think innovation should focus more. For example, a focus on *data* -- curation, measurement, consent, credit, compensation, safety -- would deeply improve technology for everyone.
7. The transparency this release provides is massive for people who want to *learn* about LLMs. For the next generation of technologists to advance over the current, they MUST be able to learn about what's happening now. (cont...)
0. Common ground setting: OpenAI is the ChatGPT people. An โopen sourceโ model is one whose weights are available โ that means the model can be โyoursโ.
1. You donโt have to interact with the company directly, nor give them your interactions, to use the system. The company can't "surveil" you.
2. You can evaluate the unique contributions of their SOTA model much more rigorously than you can when there are collections of models+code behind a closed API. You can find out specifically what the model can and can't do.
3. And you can directly customize it for whatever you'd like. Fine-tuning, wherein you give the model data that's tailored to your use cases and train it some more on that data, is trivial* when you have the model weights.
*Provided you have the compute.
4. You can directly benchmark whatever you'd like. Biases? Energy usage? Strengths/weaknesses? Go for it. You wants it you gots it--this transparency helps people understand SOTA *in general*, not just for this model, but points to, e.g., what's going on with closed Google models as well.
5. One of the most powerful things about "openness" that I've learned is that it cultivates ecosystems of collaborators building on top of one another's brilliance to make systems that are significantly better than they would be if created in isolation.
But, caveat wrt my own philosophy...
6. I do not take it as a given that advancing LLMs is good, and have a lot more to say wrt where I think innovation should focus more. For example, a focus on *data* -- curation, measurement, consent, credit, compensation, safety -- would deeply improve technology for everyone.
7. The transparency this release provides is massive for people who want to *learn* about LLMs. For the next generation of technologists to advance over the current, they MUST be able to learn about what's happening now. (cont...)
- 1 reply
alielfilali01ย
posted
an
update
3 months ago
Post
720
Guys WTH is "yofo-*" ???
Most OpenAI staff associated with the openai/gpt-oss-68911959590a1634ba11c7a4 release are affiliated to dozens of yofo orgs ...
i.e
yofo-wildflower
Some HF folks as well ๐
Most OpenAI staff associated with the openai/gpt-oss-68911959590a1634ba11c7a4 release are affiliated to dozens of yofo orgs ...
i.e
yofo-wildflower
Some HF folks as well ๐
Post
3586
๐ซก I am the first and only one to like the French Tax Code Dataset
that's it , that's the post
find the dataset here : louisbrulenaudet/code-impots
follow : @louisbrulenaudet
that's it , that's the post
find the dataset here : louisbrulenaudet/code-impots
follow : @louisbrulenaudet
- 3 replies
Post
493
๐ค ๐พ Thanks so much to BBC News and the stellar Suranjana Tewari for having me on to talk about US <โ> China relationship in AI, and what it means for AI ethics.
Post
4182
๐๐ถ๐ฟ๐๐ ๐๐ฃ๐๐ ๐ ๐ผ๐ฑ๐ฒ๐น ๐๐ถ๐๐ต ๐๐จ ๐๐ฎ๐๐ฎ ๐ง๐ฟ๐ฎ๐ป๐๐ฝ๐ฎ๐ฟ๐ฒ๐ป๐ฐ๐ ๐ง๐ฒ๐บ๐ฝ๐น๐ฎ๐๐ฒ? ๐ช๐บ
With the release of the EU data transparency template this week, we finally got to see one of the most meaningful artifacts to come out of the AI Act implementation so far (haven't you heard? AI's all about the data! ๐๐)
The impact of the template will depend on how effectively it establishes a minimum meaningful transparency standard for companies that don't otherwise offer any transparency into their handling of e.g. personal data or (anti?-)competitive practices in commercial licensing - we'll see how those play out as new models are released after August 2nd ๐
In the meantime, I wanted to see how the template works for a fully open-source + commercially viable model, so I filled it out for the SmolLM3 - which my colleagues at Hugging Face earlier this month ๐ค ICYMI, it's fully open-source with 3B parameters and performance matching the best similar-size models (I've switched all my local apps from Qwen3 to it, you should too ๐ก)
Verdict: congrats to the European Commission AI Office for making it so straightforward! Fully open and transparent models remain a cornerstone of informed regulation and governance, but the different organizational needs of their developers aren't always properly accounted for in new regulation. In this case, it took me all of two hours to fill out and publish the template (including reading the guidelines) - so kudos for making it feasible for smaller and distributed organizations ๐ Definitely a step forward for transparency ๐
To learn more have a look at:
- The SmolLM3 model: HuggingFaceTB/SmolLM3-3B
- Its filled out Public Summary of Training Content: hfmlsoc/smollm3-eu-data-transparency
- And if you're interested, some previous remarks on regulatory minimum meaningful standards for data disclosure: https://huggingface.co/blog/yjernite/naiac-data-transparency
With the release of the EU data transparency template this week, we finally got to see one of the most meaningful artifacts to come out of the AI Act implementation so far (haven't you heard? AI's all about the data! ๐๐)
The impact of the template will depend on how effectively it establishes a minimum meaningful transparency standard for companies that don't otherwise offer any transparency into their handling of e.g. personal data or (anti?-)competitive practices in commercial licensing - we'll see how those play out as new models are released after August 2nd ๐
In the meantime, I wanted to see how the template works for a fully open-source + commercially viable model, so I filled it out for the SmolLM3 - which my colleagues at Hugging Face earlier this month ๐ค ICYMI, it's fully open-source with 3B parameters and performance matching the best similar-size models (I've switched all my local apps from Qwen3 to it, you should too ๐ก)
Verdict: congrats to the European Commission AI Office for making it so straightforward! Fully open and transparent models remain a cornerstone of informed regulation and governance, but the different organizational needs of their developers aren't always properly accounted for in new regulation. In this case, it took me all of two hours to fill out and publish the template (including reading the guidelines) - so kudos for making it feasible for smaller and distributed organizations ๐ Definitely a step forward for transparency ๐
To learn more have a look at:
- The SmolLM3 model: HuggingFaceTB/SmolLM3-3B
- Its filled out Public Summary of Training Content: hfmlsoc/smollm3-eu-data-transparency
- And if you're interested, some previous remarks on regulatory minimum meaningful standards for data disclosure: https://huggingface.co/blog/yjernite/naiac-data-transparency
Post
805
๐ Hey there folks,
just submitted my plugin idea to the G-Assist Plugin Hackathon by @nvidia . Check it out, it's a great way to use a local SLA model on a windows machine to easily and locally get things done ! https://github.com/NVIDIA/G-Assist
just submitted my plugin idea to the G-Assist Plugin Hackathon by @nvidia . Check it out, it's a great way to use a local SLA model on a windows machine to easily and locally get things done ! https://github.com/NVIDIA/G-Assist
Post
624
๐๐ปโโ๏ธ Hey there folks ,
Yesterday , Nvidia released a reasoning model that beats o3 on science, math and coding !
Today you can try it out here : Tonic/Nvidia-OpenReasoning
hope you like it !
Yesterday , Nvidia released a reasoning model that beats o3 on science, math and coding !
Today you can try it out here : Tonic/Nvidia-OpenReasoning
hope you like it !