Hugging Face's logo
Doron Adler's picture
121 86 905

Doron Adler PRO

Norod78
Sa-m's profile picture ucalyptus's profile picture abep's profile picture
·
https://linktr.ee/Norod78

AI & ML interests

Fooling around with Generative machine learning models.

Recent Activity

liked a Space 1 day ago
elismasilva/gradio_tokenizertextbox
reacted to John6666's post with 👍 1 day ago
If your Space stops working after restarting mainly for the last 5 days (https://discuss.huggingface.co/t/my-space-suddenly-went-offline-the-cpu-cannot-restart/151121/22), try some of following. 1. Add `pydantic==2.10.6` to `requirements.txt` or upgrade Gradio to the latest version. 2. Upgrade PyTorch to 2.2.0 or later (`torch>=2.2.0` for Zero GPU space). 3. Fix Transformers to 4.49.0 or earlier (`transformers<=4.49.0`for spaces using Transformers or Diffusers). 4. Fix `huggingface_hub` to the old version (`huggingface_hub==0.25.2` for if an error like `cached_download` is not available occurs or inference does not work properly) 5. Specifying `WORKDIR` in `Dockerfile` may cause the application to fail to start with error 137. (Docker Spaces, https://discuss.huggingface.co/t/error-code-137-cache-error/152177) About `pydantic==2.10.6`: https://discuss.huggingface.co/t/error-no-api-found/146226 https://discuss.huggingface.co/t/internal-server-error-bool-not-iterable/149494 Edit: Zero GPU space has been upgraded from A100 to H200. This is likely the reason why older versions of PyTorch are no longer supported. In fact, an error message to that effect was displayed. https://huggingface.co/spaces/zero-gpu-explorers/README/discussions/163
posted an update 2 days ago
Multilingual Tokenization Showdown Analyzing 12 LLM Tokenizers Across 204 Languages. First, I've created a dataset with Wikipedia's "Cat" article text in 272 languages: https://huggingface.co/datasets/Norod78/WikiCat-Multilingual For each language entry with at least 100 words, I tokenized the text using 12 tokenizers and calculated the "Characters per token" ratio and "Word per token" ratio. The higher this ratio is, the more information each token represents on average for that language (and perhaps allowing the llm to potentially learn more per-parameter if trained on a dataset of that language). You can see a slideshow summary of the results here: https://norod.github.io/wikicat-tokenizer-eval/tokenizer-slideshow.html I hope I interpreted the results correctly, I've made the code available on GitHub so you can re-create the raw results jsonl with this repo: https://github.com/Norod/wikicat-tokenizer-eval Post on X: https://x.com/Norod78/status/1984366900550266999
View all activity

Organizations

Spaces-explorers's profile picture Gradio-Blocks-Party's profile picture Yam Peleg's profile picture ZeroGPU Explorers's profile picture Mixtiles's profile picture Social Post Explorers's profile picture Hugging Face Discord Community's profile picture Endless Technologies Ltd. 's profile picture