Add disclaimer
Browse files
    	
        README.md
    CHANGED
    
    | @@ -1,70 +1,74 @@ | |
| 1 | 
            -
            ---
         | 
| 2 | 
            -
            license: apache-2.0
         | 
| 3 | 
            -
            language:
         | 
| 4 | 
            -
            - en
         | 
| 5 | 
            -
            ---
         | 
| 6 | 
            -
             | 
| 7 | 
            -
            # Mixtral-8x7b-Instruct-v0.1-int8-ov
         | 
| 8 | 
            -
             | 
| 9 | 
            -
             * Model creator: [Mistral AI](https://huggingface.co/mistralai)
         | 
| 10 | 
            -
             * Original model: [Mixtral 8X7B Instruct v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
         | 
| 11 | 
            -
             | 
| 12 | 
            -
            ## Description
         | 
| 13 | 
            -
             | 
| 14 | 
            -
            This is [Mixtral-8x7b-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model converted to the [OpenVINO™ IR](https://docs.openvino.ai/2024/documentation/openvino-ir-format.html) (Intermediate Representation) format with weights compressed to INT8 by [NNCF](https://github.com/openvinotoolkit/nncf).
         | 
| 15 | 
            -
             | 
| 16 | 
            -
            ## Quantization Parameters
         | 
| 17 | 
            -
             | 
| 18 | 
            -
            Weight compression was performed using `nncf.compress_weights` with the following parameters:
         | 
| 19 | 
            -
             | 
| 20 | 
            -
            * mode: **INT8_ASYM**
         | 
| 21 | 
            -
             | 
| 22 | 
            -
            For more information on quantization, check the [OpenVINO model optimization guide](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html).
         | 
| 23 | 
            -
             | 
| 24 | 
            -
            ## Compatibility
         | 
| 25 | 
            -
             | 
| 26 | 
            -
            The provided OpenVINO™ IR model is compatible with:
         | 
| 27 | 
            -
             | 
| 28 | 
            -
            * OpenVINO version 2024.0.0 and higher
         | 
| 29 | 
            -
            * Optimum Intel 1.16.0 and higher
         | 
| 30 | 
            -
             | 
| 31 | 
            -
            ## Running Model Inference
         | 
| 32 | 
            -
             | 
| 33 | 
            -
            1. Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO backend:
         | 
| 34 | 
            -
             | 
| 35 | 
            -
            ```
         | 
| 36 | 
            -
            pip install optimum[openvino]
         | 
| 37 | 
            -
            ```
         | 
| 38 | 
            -
             | 
| 39 | 
            -
            2. Run model inference:
         | 
| 40 | 
            -
             | 
| 41 | 
            -
            ```
         | 
| 42 | 
            -
            from transformers import AutoTokenizer
         | 
| 43 | 
            -
            from optimum.intel.openvino import OVModelForCausalLM
         | 
| 44 | 
            -
             | 
| 45 | 
            -
            model_id = "OpenVINO/mixtral-8x7b-instruct-v0.1-int8-ov"
         | 
| 46 | 
            -
            tokenizer = AutoTokenizer.from_pretrained(model_id)
         | 
| 47 | 
            -
            model = OVModelForCausalLM.from_pretrained(model_id)
         | 
| 48 | 
            -
             | 
| 49 | 
            -
             | 
| 50 | 
            -
            messages = [
         | 
| 51 | 
            -
                {"role": "user", "content": "What is your favourite condiment?"},
         | 
| 52 | 
            -
                {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
         | 
| 53 | 
            -
                {"role": "user", "content": "Do you have mayonnaise recipes?"}
         | 
| 54 | 
            -
            ]
         | 
| 55 | 
            -
             | 
| 56 | 
            -
            inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
         | 
| 57 | 
            -
             | 
| 58 | 
            -
            outputs = model.generate(inputs, max_new_tokens=20)
         | 
| 59 | 
            -
            print(tokenizer.decode(outputs[0], skip_special_tokens=True))
         | 
| 60 | 
            -
            ```
         | 
| 61 | 
            -
             | 
| 62 | 
            -
            For more examples and possible optimizations, refer to the [OpenVINO Large Language Model Inference Guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html).
         | 
| 63 | 
            -
             | 
| 64 | 
            -
            ## Limitations
         | 
| 65 | 
            -
             | 
| 66 | 
            -
            Check the original model card for [limitations](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1#limitations).
         | 
| 67 | 
            -
             | 
| 68 | 
            -
            ## Legal information
         | 
| 69 | 
            -
             | 
| 70 | 
            -
            The original model is distributed under [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) license. More details can be found in [original model card](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1).
         | 
|  | |
|  | |
|  | |
|  | 
|  | |
| 1 | 
            +
            ---
         | 
| 2 | 
            +
            license: apache-2.0
         | 
| 3 | 
            +
            language:
         | 
| 4 | 
            +
            - en
         | 
| 5 | 
            +
            ---
         | 
| 6 | 
            +
             | 
| 7 | 
            +
            # Mixtral-8x7b-Instruct-v0.1-int8-ov
         | 
| 8 | 
            +
             | 
| 9 | 
            +
             * Model creator: [Mistral AI](https://huggingface.co/mistralai)
         | 
| 10 | 
            +
             * Original model: [Mixtral 8X7B Instruct v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
         | 
| 11 | 
            +
             | 
| 12 | 
            +
            ## Description
         | 
| 13 | 
            +
             | 
| 14 | 
            +
            This is [Mixtral-8x7b-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model converted to the [OpenVINO™ IR](https://docs.openvino.ai/2024/documentation/openvino-ir-format.html) (Intermediate Representation) format with weights compressed to INT8 by [NNCF](https://github.com/openvinotoolkit/nncf).
         | 
| 15 | 
            +
             | 
| 16 | 
            +
            ## Quantization Parameters
         | 
| 17 | 
            +
             | 
| 18 | 
            +
            Weight compression was performed using `nncf.compress_weights` with the following parameters:
         | 
| 19 | 
            +
             | 
| 20 | 
            +
            * mode: **INT8_ASYM**
         | 
| 21 | 
            +
             | 
| 22 | 
            +
            For more information on quantization, check the [OpenVINO model optimization guide](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html).
         | 
| 23 | 
            +
             | 
| 24 | 
            +
            ## Compatibility
         | 
| 25 | 
            +
             | 
| 26 | 
            +
            The provided OpenVINO™ IR model is compatible with:
         | 
| 27 | 
            +
             | 
| 28 | 
            +
            * OpenVINO version 2024.0.0 and higher
         | 
| 29 | 
            +
            * Optimum Intel 1.16.0 and higher
         | 
| 30 | 
            +
             | 
| 31 | 
            +
            ## Running Model Inference
         | 
| 32 | 
            +
             | 
| 33 | 
            +
            1. Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO backend:
         | 
| 34 | 
            +
             | 
| 35 | 
            +
            ```
         | 
| 36 | 
            +
            pip install optimum[openvino]
         | 
| 37 | 
            +
            ```
         | 
| 38 | 
            +
             | 
| 39 | 
            +
            2. Run model inference:
         | 
| 40 | 
            +
             | 
| 41 | 
            +
            ```
         | 
| 42 | 
            +
            from transformers import AutoTokenizer
         | 
| 43 | 
            +
            from optimum.intel.openvino import OVModelForCausalLM
         | 
| 44 | 
            +
             | 
| 45 | 
            +
            model_id = "OpenVINO/mixtral-8x7b-instruct-v0.1-int8-ov"
         | 
| 46 | 
            +
            tokenizer = AutoTokenizer.from_pretrained(model_id)
         | 
| 47 | 
            +
            model = OVModelForCausalLM.from_pretrained(model_id)
         | 
| 48 | 
            +
             | 
| 49 | 
            +
             | 
| 50 | 
            +
            messages = [
         | 
| 51 | 
            +
                {"role": "user", "content": "What is your favourite condiment?"},
         | 
| 52 | 
            +
                {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
         | 
| 53 | 
            +
                {"role": "user", "content": "Do you have mayonnaise recipes?"}
         | 
| 54 | 
            +
            ]
         | 
| 55 | 
            +
             | 
| 56 | 
            +
            inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
         | 
| 57 | 
            +
             | 
| 58 | 
            +
            outputs = model.generate(inputs, max_new_tokens=20)
         | 
| 59 | 
            +
            print(tokenizer.decode(outputs[0], skip_special_tokens=True))
         | 
| 60 | 
            +
            ```
         | 
| 61 | 
            +
             | 
| 62 | 
            +
            For more examples and possible optimizations, refer to the [OpenVINO Large Language Model Inference Guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html).
         | 
| 63 | 
            +
             | 
| 64 | 
            +
            ## Limitations
         | 
| 65 | 
            +
             | 
| 66 | 
            +
            Check the original model card for [limitations](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1#limitations).
         | 
| 67 | 
            +
             | 
| 68 | 
            +
            ## Legal information
         | 
| 69 | 
            +
             | 
| 70 | 
            +
            The original model is distributed under [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) license. More details can be found in [original model card](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1).
         | 
| 71 | 
            +
             | 
| 72 | 
            +
            ## Disclaimer
         | 
| 73 | 
            +
             | 
| 74 | 
            +
            Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.
         | 

