Spaces:

ggunio
/

intelligent-tokenizer-v6-demo

Running

App Files Files Community

intelligent-tokenizer-v6-demo

425 MB

2 contributors

History: 31 commits

ggunio's picture

Fix UTF-8 safe chunking, token boundary visualization, and embedding display

0250815 2 months ago

core
Fix import error by adding core module files 2 months ago
src
Upload src/core/byte_tokenizer_v6.py with huggingface_hub 3 months ago
.gitattributes

1.52 kB

initial commit 3 months ago
README.md

8.11 kB

Trigger rebuild after model upload 2 months ago
VERSION_COMPARISON.md

8.54 kB

Update to B2NL v6.1.2 POC - 18.6:1 compression with 6 languages (Korean, English, Chinese, Japanese, Spanish, Arabic) 2 months ago
app.py

20.5 kB

Fix UTF-8 safe chunking, token boundary visualization, and embedding display 2 months ago
config.json

394 Bytes

Upload config.json with huggingface_hub 3 months ago
demo_poc.py

9.88 kB

Upload demo_poc.py with huggingface_hub 3 months ago
inference.py

9.67 kB

Upload inference.py with huggingface_hub 3 months ago
pytorch_model.bin
Detected Pickle imports (3)
- "torch.FloatStorage",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict"
What is a pickle import?
425 MB
xet

Upload pytorch_model.bin with huggingface_hub 3 months ago
requirements.txt

65 Bytes

Fix reconstruction issue by downloading checkpoint from HF model repo 2 months ago
test_app.py

6.05 kB

Update to B2NL v6.1.2 POC - 18.6:1 compression with 6 languages (Korean, English, Chinese, Japanese, Spanish, Arabic) 2 months ago