Spaces:

ggunio
/

intelligent-tokenizer-v6-demo

Sleeping

App Files Files Community

intelligent-tokenizer-v6-demo

338 kB

2 contributors

History: 47 commits

ggunio's picture

Fix: Increase max_length to 256 for proper text reconstruction

da2970e verified about 1 month ago

core
Fix: Increase max_length to 256 for proper text reconstruction about 1 month ago
src
Upload src/core/byte_tokenizer_v6.py with huggingface_hub about 2 months ago
.gitattributes

1.52 kB

initial commit about 2 months ago
README.md

987 Bytes

Update README with v6.2.1 info and author about 1 month ago
VERSION_COMPARISON.md

8.54 kB

Update to B2NL v6.1.2 POC - 18.6:1 compression with 6 languages (Korean, English, Chinese, Japanese, Spanish, Arabic) about 1 month ago
app.py

18.2 kB

Fix: Correct embedding dimension (1280-dim, not 768-dim) about 1 month ago
config.json

394 Bytes

Upload config.json with huggingface_hub about 2 months ago
demo_poc.py

9.88 kB

Upload demo_poc.py with huggingface_hub about 2 months ago
inference.py

6.29 kB

Upload inference.py with huggingface_hub about 1 month ago
requirements.txt

103 Bytes

Upload requirements.txt with huggingface_hub about 1 month ago
test_app.py

6.05 kB

Update to B2NL v6.1.2 POC - 18.6:1 compression with 6 languages (Korean, English, Chinese, Japanese, Spanish, Arabic) about 1 month ago