Update README.md
Browse files
README.md
CHANGED
|
@@ -59,10 +59,9 @@ def benchmark_fn(f, *args, **kwargs):
|
|
| 59 |
torchao.quantization.utils.recommended_inductor_config_setter()
|
| 60 |
quantized_model = torch.compile(quantized_model, mode="max-autotune")
|
| 61 |
print(f"{save_to} model:", benchmark_fn(quantized_model.generate, **inputs, max_new_tokens=128))
|
| 62 |
-
|
| 63 |
# Model Quality
|
| 64 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
| 65 |
-
```
|
| 66 |
|
| 67 |
# Installing the nightly version to get most recent updates
|
| 68 |
```
|
|
@@ -119,7 +118,7 @@ python benchmarks/benchmark_latency.py --input-len 256 --output-len 256 --model
|
|
| 119 |
|
| 120 |
# benchmark_serving
|
| 121 |
|
| 122 |
-
We also benchmarked the throughput
|
| 123 |
|
| 124 |
## baseline
|
| 125 |
Server:
|
|
|
|
| 59 |
torchao.quantization.utils.recommended_inductor_config_setter()
|
| 60 |
quantized_model = torch.compile(quantized_model, mode="max-autotune")
|
| 61 |
print(f"{save_to} model:", benchmark_fn(quantized_model.generate, **inputs, max_new_tokens=128))
|
| 62 |
+
```
|
| 63 |
# Model Quality
|
| 64 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
|
|
|
| 65 |
|
| 66 |
# Installing the nightly version to get most recent updates
|
| 67 |
```
|
|
|
|
| 118 |
|
| 119 |
# benchmark_serving
|
| 120 |
|
| 121 |
+
We also benchmarked the throughput in a serving environment.
|
| 122 |
|
| 123 |
## baseline
|
| 124 |
Server:
|