Update README.md (#224)

- Update README.md (fe95b00fbbd8d765d66350eba58a2c428f379ab9)

README.md

@@ -21,46 +21,54 @@ widget:
 > [!CAUTION]
 > ⚠️ 
 > The `transformers` tokenizer might give incorrect results as it has not been tested by the Mistral team. To make sure that your encoding and decoding is correct, please use mistral-common as shown below: 
-> 
-> ```py
-> from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
-> from mistral_common.protocol.instruct.messages import UserMessage
-> from mistral_common.protocol.instruct.request import ChatCompletionRequest
-> 
-> mistral_models_path = "MISTRAL_MODELS_PATH"
-> 
-> tokenizer = MistralTokenizer.v1()
-> 
-> completion_request = ChatCompletionRequest(messages=[UserMessage(content="Explain Machine Learning to me in a nutshell.")])
-> 
-> tokens = tokenizer.encode_chat_completion(completion_request).tokens
-> 
-> ## Inference with `mistral_inference`
-> 
-> from mistral_inference.model import Transformer
-> from mistral_inference.generate import generate
-> 
-> model = Transformer.from_folder(mistral_models_path)
-> out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
-> result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
-> 
-> print(result)
-> 
-> ## Inference with hugging face `transformers`
-> 
-> from transformers import AutoModelForCausalLM, AutoTokenizer
-> 
-> device = "cuda"
-> model = AutoModelForCausalLM.from_pretrained(mistralai/Mixtral-8x7B-Instruct-v0.1)
-> model.to(device)
-> 
-> generated_ids = model.generate(tokens, max_new_tokens=1000, do_sample=True)
-> decoded = tokenizer.batch_decode(generated_ids)
-> print(decoded[0])
-> ```
-> 
+
+
+```py
+from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
+from mistral_common.protocol.instruct.messages import UserMessage
+from mistral_common.protocol.instruct.request import ChatCompletionRequest
+ 
+mistral_models_path = "MISTRAL_MODELS_PATH"
+ 
+tokenizer = MistralTokenizer.v1()
+ 
+completion_request = ChatCompletionRequest(messages=[UserMessage(content="Explain Machine Learning to me in a nutshell.")])
+ 
+tokens = tokenizer.encode_chat_completion(completion_request).tokens
+```
+ 
+## Inference with `mistral_inference`
+ 
+ ```py
+from mistral_inference.model import Transformer
+from mistral_inference.generate import generate
+ 
+model = Transformer.from_folder(mistral_models_path)
+out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
+
+result = tokenizer.decode(out_tokens[0])
+
+print(result)
+```
+
+## Inference with hugging face `transformers`
+ 
+```py
+from transformers import AutoModelForCausalLM
+ 
+model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1")
+model.to("cuda")
+ 
+generated_ids = model.generate(tokens, max_new_tokens=1000, do_sample=True)
+
+# decode with mistral tokenizer
+result = tokenizer.decode(generated_ids[0].tolist())
+print(result)
+```
+
+> [!TIP]
 > PRs to correct the transformers tokenizer so that it gives 1-to-1 the same results as the mistral-common reference implementation are very welcome!
->             
+     
         
 ---
 The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. The Mixtral-8x7B outperforms Llama 2 70B on most benchmarks we tested.