Spaces:

mponty
/

some_tokenization

Sleeping

mponty commited on Dec 11, 2024

Commit

2d37735

verified ·

1 Parent(s): 4d09e88

Update app.py

Files changed (1) hide show

app.py CHANGED Viewed

@@ -29,7 +29,7 @@ tokenizer = st.selectbox(
         "White Space",
         "Qwen2.5 Tokenizer",
         "RuAdapt Tokenizer",
-        "Aya-Expanse Tokenizer"
         "Byte Pair Encoding (Open AI GPT-4o)",
     ),
     index=None,

         "White Space",
         "Qwen2.5 Tokenizer",
         "RuAdapt Tokenizer",
+        "Aya-Expanse Tokenizer",
         "Byte Pair Encoding (Open AI GPT-4o)",
     ),
     index=None,