Tokenizer Eval

tokenizer-eval's activity

• All
• Models
• Datasets
• Spaces
• Papers
• Collections
• Community
• Posts
• Articles

christopher 

authored a paper 10 days ago

The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models

Paper • 2510.13996 • Published 13 days ago • 6

christopher 

posted an update 21 days ago

view post

Post

418

Something very cool is cooking at Lichess

See translation

• 

• 
• 1 reply

·

👀

1

1

+

Reply

christopher 

posted an update 11 months ago

view post

Post

2103

The folks at Foursquare released a dataset of 104.5 million places of interest ( foursquare/fsq-os-places) and here's all of them on a plot

See translation

• 

• 
• 
• 
• 4 replies

·

🔥

5

5

🚀

1

1

😔

1

1

+

Reply

christopher 

posted an update 11 months ago

view post

Post

2440

The Lichess database of games, puzzles, and engine evaluations is now on the Hub: Lichess

Billions of chess data points to download, query, and stream and we're excited to see what you'll build with it! ♟️ 🤗

- https://huggingface.co/collections/Lichess/positions-datasets-66f50837db5cd3287d60d489
- https://huggingface.co/collections/Lichess/games-datasets-66f508df78f4b43e1bb2d353

See translation

👍

7

7

❤️

2

2

🔥

1

1

+

Reply

christopher 

posted an update about 1 year ago

view post

Post

1341

4 million chess puzzles

• 

❤️

6

6

👀

3

3

+

Reply

christopher 

authored a paper over 1 year ago

StarCoder 2 and The Stack v2: The Next Generation

Paper • 2402.19173 • Published Feb 29, 2024 • 148

christopher 

posted an update over 1 year ago

view post

Post

Data map of the languages of https://huggingface.co/datasets/CohereForAI/aya_dataset

• 

• 
• 1 reply

·

❤️

12

12

+

Reply

christopher 

posted an update over 1 year ago

view post

Post

TIL: EleutherAI/pile is on Wikipedia: https://en.wikipedia.org/wiki/The_Pile_(dataset)

🤯

5

5

🤗

4

4

❤️

1

1

+

Reply

christopher 

authored a paper almost 2 years ago

GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration

Paper • 2306.01481 • Published Jun 2, 2023 • 2

christopher 

authored 7 papers over 2 years ago

StarCoder: may the source be with you!

Paper • 2305.06161 • Published May 9, 2023 • 31

Stable Bias: Analyzing Societal Representations in Diffusion Models

Paper • 2303.11408 • Published Mar 20, 2023 • 1

BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model

Paper • 2212.04960 • Published Dec 9, 2022 • 1

Towards Openness Beyond Open Access: User Journeys through 3 Open AI Collaboratives

Paper • 2301.08488 • Published Jan 20, 2023 • 1

The ROOTS Search Tool: Data Transparency for LLMs

Paper • 2302.14035 • Published Feb 27, 2023

SantaCoder: don't reach for the stars!

Paper • 2301.03988 • Published Jan 9, 2023 • 7

Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face

Paper • 2302.14534 • Published Feb 28, 2023 • 1