Update README.md
Browse files
    	
        README.md
    CHANGED
    
    | @@ -18,7 +18,7 @@ language: | |
| 18 |  | 
| 19 |  | 
| 20 |  | 
| 21 | 
            -
            Calibrated with  | 
| 22 |  | 
| 23 |  | 
| 24 | 
             
            # Inference with vLLM
         | 
| @@ -219,7 +219,7 @@ We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-h | |
| 219 | 
             
            | Benchmark                        |                        |                             |                                 |
         | 
| 220 | 
             
            |----------------------------------|------------------------|-----------------------------|---------------------------------|
         | 
| 221 | 
             
            |                                  | google/gemma-3-12b-it  | pytorch/gemma-3-12b-it-INT4 | pytorch/gemma-3-12b-it-AWQ-INT4 |
         | 
| 222 | 
            -
            |  | 
| 223 |  | 
| 224 |  | 
| 225 | 
             
            <details>
         | 
|  | |
| 18 |  | 
| 19 |  | 
| 20 |  | 
| 21 | 
            +
            Calibrated with 30 samples of `mmlu_philosophy`, got eval accuracy of 76.86, while gemma-3-12b-it-INT4 is 75.56, and bfloat16 baseline is 79.10
         | 
| 22 |  | 
| 23 |  | 
| 24 | 
             
            # Inference with vLLM
         | 
|  | |
| 219 | 
             
            | Benchmark                        |                        |                             |                                 |
         | 
| 220 | 
             
            |----------------------------------|------------------------|-----------------------------|---------------------------------|
         | 
| 221 | 
             
            |                                  | google/gemma-3-12b-it  | pytorch/gemma-3-12b-it-INT4 | pytorch/gemma-3-12b-it-AWQ-INT4 |
         | 
| 222 | 
            +
            | philosophy                       | 79.10                  |     75.56                   | 76.85                            |
         | 
| 223 |  | 
| 224 |  | 
| 225 | 
             
            <details>
         | 
