jerryzh168 commited on
Commit
51c1549
·
verified ·
1 Parent(s): c568eae

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -18,7 +18,7 @@ language:
18
 
19
 
20
 
21
- Calibrated with 10 samples of `mmlu_abstract_algebra`, got eval accuracy of 42, while gemma-3-12b-it-INT4 is 41, and bfloat16 baseline is 43
22
 
23
 
24
  # Inference with vLLM
@@ -219,7 +219,7 @@ We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-h
219
  | Benchmark | | | |
220
  |----------------------------------|------------------------|-----------------------------|---------------------------------|
221
  | | google/gemma-3-12b-it | pytorch/gemma-3-12b-it-INT4 | pytorch/gemma-3-12b-it-AWQ-INT4 |
222
- | professional_law | TODO | 54.24 | TODO |
223
 
224
 
225
  <details>
 
18
 
19
 
20
 
21
+ Calibrated with 30 samples of `mmlu_philosophy`, got eval accuracy of 76.86, while gemma-3-12b-it-INT4 is 75.56, and bfloat16 baseline is 79.10
22
 
23
 
24
  # Inference with vLLM
 
219
  | Benchmark | | | |
220
  |----------------------------------|------------------------|-----------------------------|---------------------------------|
221
  | | google/gemma-3-12b-it | pytorch/gemma-3-12b-it-INT4 | pytorch/gemma-3-12b-it-AWQ-INT4 |
222
+ | philosophy | 79.10 | 75.56 | 76.85 |
223
 
224
 
225
  <details>