Update README.md
Browse files
README.md
CHANGED
|
@@ -17,16 +17,14 @@ base_model:
|
|
| 17 |
pipeline_tag: text-generation
|
| 18 |
---
|
| 19 |
|
| 20 |
-
#
|
|
|
|
|
|
|
| 21 |
```
|
| 22 |
pip install git+https://github.com/huggingface/transformers@main
|
| 23 |
pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu126
|
| 24 |
-
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
|
| 25 |
```
|
| 26 |
|
| 27 |
-
Also need to install lm-eval from source: https://github.com/EleutherAI/lm-evaluation-harness#install
|
| 28 |
-
|
| 29 |
-
# Quantization Recipe
|
| 30 |
We used following code to get the quantized model:
|
| 31 |
|
| 32 |
```
|
|
@@ -119,6 +117,8 @@ Hello! As an AI, I don't have consciousness in the way humans do, but I am fully
|
|
| 119 |
|
| 120 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
| 121 |
|
|
|
|
|
|
|
| 122 |
## baseline
|
| 123 |
```
|
| 124 |
lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks hellaswag --device cuda:0 --batch_size 64
|
|
|
|
| 17 |
pipeline_tag: text-generation
|
| 18 |
---
|
| 19 |
|
| 20 |
+
# Quantization Recipe
|
| 21 |
+
|
| 22 |
+
First need to install the required packages:
|
| 23 |
```
|
| 24 |
pip install git+https://github.com/huggingface/transformers@main
|
| 25 |
pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu126
|
|
|
|
| 26 |
```
|
| 27 |
|
|
|
|
|
|
|
|
|
|
| 28 |
We used following code to get the quantized model:
|
| 29 |
|
| 30 |
```
|
|
|
|
| 117 |
|
| 118 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
| 119 |
|
| 120 |
+
Need to install lm-eval from source: https://github.com/EleutherAI/lm-evaluation-harness#install
|
| 121 |
+
|
| 122 |
## baseline
|
| 123 |
```
|
| 124 |
lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks hellaswag --device cuda:0 --batch_size 64
|