Update README.md
Browse files
README.md
CHANGED
|
@@ -62,21 +62,27 @@ print(f"{save_to} model:", benchmark_fn(quantized_model.generate, **inputs, max_
|
|
| 62 |
|
| 63 |
# Model Quality
|
| 64 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
| 65 |
-
|
| 66 |
```
|
|
|
|
| 67 |
# Installing the nightly version to get most recent updates
|
|
|
|
| 68 |
pip install git+https://github.com/EleutherAI/lm-evaluation-harness
|
|
|
|
| 69 |
|
| 70 |
# baseline
|
|
|
|
| 71 |
lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks hellaswag --device cuda:0 --batch_size 8
|
|
|
|
| 72 |
|
| 73 |
# int4wo-hqq
|
|
|
|
| 74 |
lm_eval --model hf --model_args pretrained=jerryzh168/phi4-mini-int4wo-hqq --tasks hellaswag --device cuda:0 --batch_size 8
|
| 75 |
```
|
| 76 |
|
| 77 |
`TODO: more complete eval results`
|
| 78 |
|
| 79 |
-
|
|
|
|
| 80 |
|----------------------------------|-------------|-------------------|
|
| 81 |
| | Phi-4 mini-Ins | phi4-mini-int4wo |
|
| 82 |
| **Popular aggregated benchmark** | | |
|
|
@@ -91,12 +97,13 @@ lm_eval --model hf --model_args pretrained=jerryzh168/phi4-mini-int4wo-hqq --tas
|
|
| 91 |
Our int4wo is only optimized for batch size 1, so we'll only benchmark the batch size 1 performance with vllm.
|
| 92 |
For batch size N, please see our [gemlite checkpoint](https://huggingface.co/jerryzh168/phi4-mini-int4wo-gemlite).
|
| 93 |
|
| 94 |
-
```
|
| 95 |
# Install latest vllm to get the most recent changes
|
|
|
|
| 96 |
pip install git+https://github.com/vllm-project/vllm.git
|
|
|
|
| 97 |
|
| 98 |
# Download dataset
|
| 99 |
-
Download sharegpt dataset: wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
|
| 100 |
|
| 101 |
Other datasets can be found in: https://github.com/vllm-project/vllm/tree/main/benchmarks
|
| 102 |
# benchmark_latency
|
|
|
|
| 62 |
|
| 63 |
# Model Quality
|
| 64 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
|
|
|
| 65 |
```
|
| 66 |
+
|
| 67 |
# Installing the nightly version to get most recent updates
|
| 68 |
+
```
|
| 69 |
pip install git+https://github.com/EleutherAI/lm-evaluation-harness
|
| 70 |
+
```
|
| 71 |
|
| 72 |
# baseline
|
| 73 |
+
```
|
| 74 |
lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks hellaswag --device cuda:0 --batch_size 8
|
| 75 |
+
```
|
| 76 |
|
| 77 |
# int4wo-hqq
|
| 78 |
+
```
|
| 79 |
lm_eval --model hf --model_args pretrained=jerryzh168/phi4-mini-int4wo-hqq --tasks hellaswag --device cuda:0 --batch_size 8
|
| 80 |
```
|
| 81 |
|
| 82 |
`TODO: more complete eval results`
|
| 83 |
|
| 84 |
+
|
| 85 |
+
| Benchmark | | |
|
| 86 |
|----------------------------------|-------------|-------------------|
|
| 87 |
| | Phi-4 mini-Ins | phi4-mini-int4wo |
|
| 88 |
| **Popular aggregated benchmark** | | |
|
|
|
|
| 97 |
Our int4wo is only optimized for batch size 1, so we'll only benchmark the batch size 1 performance with vllm.
|
| 98 |
For batch size N, please see our [gemlite checkpoint](https://huggingface.co/jerryzh168/phi4-mini-int4wo-gemlite).
|
| 99 |
|
|
|
|
| 100 |
# Install latest vllm to get the most recent changes
|
| 101 |
+
```
|
| 102 |
pip install git+https://github.com/vllm-project/vllm.git
|
| 103 |
+
```
|
| 104 |
|
| 105 |
# Download dataset
|
| 106 |
+
Download sharegpt dataset: `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json`
|
| 107 |
|
| 108 |
Other datasets can be found in: https://github.com/vllm-project/vllm/tree/main/benchmarks
|
| 109 |
# benchmark_latency
|