Update tokenizer_config.json
#14
by
awni
- opened
This fixes the tokenizer to work with Transformers V5 otherwise you get the incorrect normalization.
See https://github.com/ml-explore/mlx-lm/issues/725 and https://github.com/huggingface/transformers/issues/43122 for relevant discussion.
Just cross commenting for viz, https://huggingface.co/MiniMaxAI/MiniMax-M2.1/discussions/15#6960e51552c999fa2d0c464d
We updated the logic so that should no longer be necessary (we auto fallback to tokenizerbackend ourselves)