Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Stragegy

  • Quantized Layers: All linear layers excluding "lm_head"
  • Weight: Auto Mixed Precision quantized by Quark, each weight has either quantization scheme in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • Activation: Auto Mixed Precision quantized by Quark, each activation input has the same quantization scheme with weight, i.e., in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • KV Cache: FP8 symmetric per-tensor

• Quick Start

1. Download and install Quark
2. [TODO] We will provide example script(s) to run auto mixed precision (AMP) quantizations later.

Deployment

The Quark quantized Auto Mixed Precision (AMP) models are now supported to be easily deployed in vLLM backend (vLLM-compatible).

Evaluation

The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.

Evaluation scores

Quant scheme	arc challenge (↑) (acc)		gsm8k (↑) (strict-match)		mmlu (↑) (acc)		winogrande (↑) (acc)
	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate
BF16	0.5597	100.0%	0.8552	100.0%	0.7296	100.0%	0.6780	100.0%
FP8	0.5563	99.4%	0.8461	98.9%	0.7282	99.8%	0.6654	98.1%
AMP	0.5154	92.1%	0.8446	98.8%	0.7171	98.3%	0.6559	96.7%
MXFP4	0.4863	86.9%	0.8112	94.9%	0.6883	94.3%	0.6504	95.9%

License

Modifications Copyright(c) 2025 Advanced Micro Devices,Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.