Seems lagging behind Pangu-1B
#4
by
hankaixyz
- opened
https://ai.gitcode.com/ascend-tribe/openPangu-Embedded-1B-V1.1
| Benchmark | Metric | MobileLLM-Pro-1B | openPangu-1B-V1.1 |
|---|---|---|---|
| General | |||
| MMLU | Acc | 44.8 | 65.08 |
| IF-Eval | Prompt Strict | 62.0 | 55.51 |
| Math & Reasoning | |||
| GSM8K | Acc | 54.0 | 82.76 |
| MATH-500 | Acc | 21.5 | 81.83 |
| Coding | |||
| MBPP | Pass@1 | 46.8 | 59.31 |
| HumanEval | Pass@1 | 59.8 | 66.66 |
Pangu-1B's outperformance in reasoning and MMLU is indeed quite striking.
Where did you get the GSM8K and MATH-500 score for MobileLLM-Pro? It's not in this repo.
Where did you get the GSM8K and MATH-500 score for MobileLLM-Pro? It's not in this repo.
from AK's blog: https://x.com/_akhaliq/status/1978916251456925757
