AngelSlim/Hy3-GPTQ-Int4

Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit.

📖 Documentation   |   🤗 Hugging Face   |   🤖 ModelScope   |   💬 WeChat

Hy3 GPTQ-Int4 Quantization

We use GPTQ 4-bit quantization to compress Hy3 to ~1/4 size with minimal accuracy loss. See the benchmark below:

Quickstart

vLLM

Build vLLM from source:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto

Start the vLLM server:

# Switch to trtllm backend to work-around mnnvl workspace size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve AngelSlim/Hy3-GPTQ-Int4 \
  --tensor-parallel-size 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy3-gptq-int4