Tokenizer Study - a shikhar-srivastava Collection

Note Model with unscaled script, bpe tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_bpe_unscaled_49152

Note Model with unscaled script, bpe tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_bpe_unscaled_65536

0.2B • Updated Aug 29 • 6

Note Model with unscaled script, bpe tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_bpe_unscaled_81920

Note Model with unscaled script, bpe tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_bpe_unscaled_98304

0.2B • Updated Aug 29 • 4

Note Model with unscaled script, bpe tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_16384

0.1B • Updated Aug 28

Note Model with unscaled script, bpe tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_32768

Note Model with unscaled script, bpe tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_49152

Note Model with unscaled script, bpe tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_65536

0.2B • Updated Aug 29 • 10

Note Model with unscaled script, bpe tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_81920

Note Model with unscaled script, bpe tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_98304

Note Model with unscaled script, bpe tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_16384

0.1B • Updated Aug 28

Note Model with unscaled script, bpe tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_32768

Note Model with unscaled script, bpe tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_49152

Note Model with unscaled script, bpe tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_65536

0.2B • Updated Aug 29 • 17

Note Model with unscaled script, bpe tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_81920

Note Model with unscaled script, bpe tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_98304

Note Model with unscaled script, bpe tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_16384

Note Model with unscaled script, bpe tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_32768

Note Model with unscaled script, bpe tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_49152

0.2B • Updated Aug 29 • 4

Note Model with unscaled script, bpe tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_65536

Note Model with unscaled script, bpe tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_81920

0.2B • Updated Aug 29 • 1

Note Model with unscaled script, bpe tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_16384

0.1B • Updated Aug 28

Note Model with unscaled script, bpe tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_32768

0.1B • Updated Aug 29 • 1

Note Model with unscaled script, bpe tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_49152

0.2B • Updated Aug 29 • 17

Note Model with unscaled script, bpe tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_65536

Note Model with unscaled script, bpe tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_81920

Note Model with unscaled script, bpe tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_98304

0.3B • Updated Aug 29 • 5

Note Model with unscaled script, bpe tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_114688

0.3B • Updated Aug 29

Note Model with unscaled script, bpe tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_bpe_unscaled_262144

Updated Aug 29

Note Model with unscaled script, bpe tokenizer, vocab size 262144

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_bpe_unscaled_114688

0.3B • Updated Aug 29

Note Model with unscaled script, bpe tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_114688

Note Model with unscaled script, bpe tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_bpe_unscaled_262144

0.5B • Updated Aug 29

Note Model with unscaled script, bpe tokenizer, vocab size 262144

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_114688

0.3B • Updated Aug 29 • 6

Note Model with unscaled script, bpe tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_unigram_unscaled_16384

Note Model with unscaled script, unigram tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_unigram_unscaled_32768

Note Model with unscaled script, unigram tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_unigram_unscaled_49152

0.2B • Updated Aug 29 • 6

Note Model with unscaled script, unigram tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_unigram_unscaled_65536

Note Model with unscaled script, unigram tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_unigram_unscaled_81920

0.2B • Updated Aug 29 • 3

Note Model with unscaled script, unigram tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_unigram_unscaled_98304

Note Model with unscaled script, unigram tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_unigram_unscaled_114688

0.3B • Updated Aug 29

Note Model with unscaled script, unigram tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_unigram_unscaled_16384

Note Model with unscaled script, unigram tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_unigram_unscaled_32768

0.2B • Updated Aug 30 • 6

Note Model with unscaled script, unigram tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_unigram_unscaled_49152

Note Model with unscaled script, unigram tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_unigram_unscaled_65536

Note Model with unscaled script, unigram tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_unigram_unscaled_81920

Note Model with unscaled script, unigram tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_unigram_unscaled_98304

Note Model with unscaled script, unigram tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_eng_latn_unigram_unscaled_114688

0.3B • Updated Aug 30

Note Model with unscaled script, unigram tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_tha_thai_bpe_unscaled_262144

0.5B • Updated Aug 29

Note Model with unscaled script, bpe tokenizer, vocab size 262144

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_unigram_unscaled_8192

97.5M • Updated Aug 30

Note Model with unscaled script, unigram tokenizer, vocab size 8192

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_unigram_unscaled_16384

Note Model with unscaled script, unigram tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_unigram_unscaled_32768

Note Model with unscaled script, unigram tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_unigram_unscaled_49152

Note Model with unscaled script, unigram tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_unigram_unscaled_65536

Note Model with unscaled script, unigram tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_unigram_unscaled_8192

97.5M • Updated Aug 30

Note Model with unscaled script, unigram tokenizer, vocab size 8192

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_unigram_unscaled_16384

Note Model with unscaled script, unigram tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_unigram_unscaled_32768

0.2B • Updated Aug 30 • 6

Note Model with unscaled script, unigram tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_unigram_unscaled_49152

Note Model with unscaled script, unigram tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_unigram_unscaled_65536

0.2B • Updated Aug 30 • 18

Note Model with unscaled script, unigram tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_unigram_unscaled_81920

Note Model with unscaled script, unigram tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_bpe_unscaled_262144

0.5B • Updated Aug 29

Note Model with unscaled script, bpe tokenizer, vocab size 262144

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_8192

97.5M • Updated Aug 30

Note Model with unscaled script, unigram tokenizer, vocab size 8192

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_16384

Note Model with unscaled script, unigram tokenizer, vocab size 16384

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_32768

Note Model with unscaled script, unigram tokenizer, vocab size 32768

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_49152

Note Model with unscaled script, unigram tokenizer, vocab size 49152

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_65536

0.2B • Updated Aug 30 • 3

Note Model with unscaled script, unigram tokenizer, vocab size 65536

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_81920

Note Model with unscaled script, unigram tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_98304

Note Model with unscaled script, bpe tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_98304

0.2B • Updated Aug 30 • 3

Note Model with unscaled script, unigram tokenizer, vocab size 98304

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_114688

0.3B • Updated Aug 30

Note Model with unscaled script, bpe tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_unigram_unscaled_114688

0.3B • Updated Aug 30

Note Model with unscaled script, unigram tokenizer, vocab size 114688

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_vie_latn_bpe_unscaled_262144

0.5B • Updated Aug 30

Note Model with unscaled script, bpe tokenizer, vocab size 262144

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_urd_arab_unigram_unscaled_81920

Note Model with unscaled script, unigram tokenizer, vocab size 81920

shikhar-srivastava/mono_gold_130m_pre_lr1e-4_amh_ethi_unigram_unscaled_98304