Spaces:

ggunio
/

intelligent-tokenizer-v6-demo

Sleeping

App Files Files Community

intelligent-tokenizer-v6-demo / README.md

ggunio

Update README with v6.2.1 info and author

0cc32d2 verified about 1 month ago

preview code

raw

history blame contribute delete

987 Bytes

A newer version of the Gradio SDK is available: 5.49.1

Upgrade

metadata

title: B2NL v6.2.1 - Byte-to-Natural Language Tokenizer 🚀
emoji: 🚀
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.19.2
app_file: app.py
pinned: true
license: apache-2.0
models:
  - ggunio/B2NL-IntelligentTokenizer-v6.2.1

B2NL v6.2.1 - Byte-to-Natural Language Tokenizer 🚀

Compress and reconstruct text with token boundaries

⚠️ IMPORTANT: Currently in AUTOREGRESSIVE MODE

Current: ~500ms inference (Teacher Forcing training)
Coming Soon (November 2025): Non-autoregressive training (<50ms)

🌟 What's New in v6.2.1

204 languages support (up from 6)
16:1 fixed compression ratio
Multi-Query Attention (8x memory reduction)
Model: ggunio/B2NL-IntelligentTokenizer-v6.2.1

Author

Jinhyun Woo

GitHub: Woojiggun/intelligent-tokenizer
Paper: Zenodo