YanissAmz/Hy3-295B-A21B-GGUF

Hy3 295B-A21B — GGUF (IQ3_XXS-UD, imatrix)

GGUF quantization of tencent/Hy3 (295B total / 21B active MoE, Apache 2.0), converted from the official FP8 checkpoint (tencent/Hy3-FP8).

Requires llama.cpp with hy-v3 architecture support. This is a brand-new architecture; support is not yet in a release. Until it is merged upstream, build llama.cpp from a branch that implements HYV3ForCausalLM. Once merged, any recent build works.

Files

file	size	recipe
Hy3-UD128-*.gguf	116.7 GB total (sharded <45 GB)	routed experts: down IQ3_S · gate IQ2_S · up IQ3_XXS — attention Q5_K · shared expert + dense FFN Q6_K · output Q6_K · embeddings Q4_K
Hy3-IQ4-UD-split-*.gguf	143.9 GB total (sharded <45 GB)	max-quality edition for dual-device rigs: routed down IQ4_XS · gate/up IQ3_S · attention/actives Q8_0
Hy3.imatrix.gguf	small	importance matrix, ~125 chunks of 512 tokens, general-purpose calibration corpus

The UD128 mix is an "unsloth-dynamic"-style asymmetric allocation. Tensors active on every token (attention, shared expert, dense layer-0 FFN, output head) keep high precision. Within the 192 routed experts — only 8 active per token, so they carry the low-bit budget — the extra bits go to the down-projection (ffn_down_exps at IQ3_S), the most quantization-sensitive expert tensor, funded by the gate projection at IQ2_S (a multiplicative mask, the most robust one); up sits in between at IQ3_XXS. Since routed experts are 97 % of the parameters, this is where the whole quality/size trade-off is decided.

Measured (wikitext-2 test, c=2048, 60 chunks, held-out from calibration): PPL 4.797 ± 0.048 for this asymmetric mix vs 4.833 ± 0.048 for a flat IQ3_XXS-experts build at the same file size (Q8_0 reference ≈ 4.3) — the redistribution is measured, not assumed.

The imatrix file is published so you can requantize any other mix directly from a Q8_0 master without redoing the (expensive) calibration pass.

Why this size

116.7 GB on disk, but the inert NextN layer (blk.80, 2 GB) is never allocated at inference, so the resident footprint is **107 GiB** — it fits fully in the GPU/unified memory of 128 GB-class machines (Strix Halo / Apple Silicon / multi-GPU rigs), no CPU offload needed. Usable unified RAM on "128 GB" machines is really 122–126 GB (firmware/OS reserve varies), and whatever else runs on the box counts against you. Comfortable defaults: 24K–32K with q8_0 KV, or 40K+ with q4_0 KV — on recent llama.cpp builds the Hadamard-rotated KV cache makes q4_0 nearly q8_0-quality, at half the size (87 KB/token vs ~160). For longer context, offload a few expert layers to CPU/second GPU.

Running

Tencent's recommended sampling: --temp 0.9 --top-p 1.0. The chat template is embedded and resolved, so plain --jinja works for chat — for tool calling, add --chat-template-file hy3-chat-template.jinja (included in this repo, see Notes).

Single device / unified memory:

llama-server -m Hy3-UD128-00001-of-00003.gguf -ngl 99 -fa on --jinja \
  -c 24576 -ctk q8_0 -ctv q8_0 --temp 0.9 --top-p 1.0   # or -c 40960 -ctk q4_0 -ctv q4_0

AMD Strix Halo / RDNA3.5 iGPU tip: run this model on the Vulkan (RADV) backend, not ROCm. Measured on a 128 GB Strix Halo (Radeon 8060S, llama-bench, same build, q4_0 KV): Vulkan 168 t/s prefill / 18.4 t/s decode vs ROCm 105 / 12.4 — and at 8K depth the ROCm flash-attention kernel collapses (26 t/s prefill vs 104 on Vulkan). ~17 t/s sustained decode in real serving at 50K context.

Split (small fast GPU + large slow memory — attention/dense/output on the fast GPU, routed experts on the big one), example CUDA + iGPU:

llama-server -m Hy3-IQ4-UD-split-00001-of-00004.gguf --device CUDA0,ROCm0 -ngl 99 -fa on --jinja \
  -c 16384 -ctk q4_0 -ctv q4_0 \
  -ot "ffn_.*_exps=ROCm0" -ot "output=CUDA0" -ot "token_embd=ROCm0" \
  --temp 0.9 --top-p 1.0

(Measured on Strix Halo iGPU + RTX 3090: 12.8 tok/s decode on the IQ4 split edition.)

Notes

• Tool calling: use the included hy3-chat-template.jinja (--jinja --chat-template-file hy3-chat-template.jinja). The embedded template renders the eos token as literal text after assistant turns; llama.cpp's tool-call autoparser bakes that literal into its induced grammar, and at generation time the real EOG token duplicates it — the grammar parse fails mid-stream (common_chat_peg_parse: unparsed peg-native output → 500 on streaming tool calls) and the eos string leaks into plain-chat content. The repo template is identical except it renders eos_token only to terminate past assistant turns in the history, never after the final (currently-generating) turn — so the induced grammar never carries a trailing eos literal, while multi-turn history stays correctly delimited. Stopping is handled by GGUF eos_token_id metadata, unaffected. Embedded metadata will be corrected in the next file revision.
• The NextN/MTP layer (blk.80) is included and measured working with llama.cpp PR #25395's --spec-type draft-mtp: on the same Strix Halo, decode goes from ~17 t/s to 24.3 t/s on code (+40%) and ~20 t/s on prose, with 91% draft acceptance at temp 0.9 (--spec-draft-p-min 0.75 is required — the MTP head is single-depth-trained and the p_min=0 default makes speculation a net slowdown). Multi-turn prompt-cache reuse is unaffected.
• Architecture naming: these files carry general.architecture = hy-v3 (this port's original naming). PR #25395 registers the arch as hy_v3 (underscore). If that PR merges as-is, rename the metadata prefix with the included gguf-arch-rename.py (metadata-only rewrite, no requant — validated: renamed files load and run MTP on the PR branch unmodified, tensor tables are identical). Renamed uploads will replace these files once upstream naming is final.
• Requantizing this yourself? The NextN layer (blk.80) has no imatrix data (it is never activated), so any very-low-bit type that requires an importance matrix — iq3_xxs, iq2_*, iq1_* — will abort on blk.80.ffn_down_exps at the end of the run. Pin that layer to an imatrix-exempt type first: --tensor-type 'blk\.80\.=q4_0' (its precision is irrelevant since it is skipped at inference). k-quants, iq3_s, and iq4_xs are unaffected.
• Quantized with a general-purpose calibration corpus — no domain-specific tuning.

Provenance / verification

• Logit parity vs transformers verified at f32 (KL = 0.0, exact top-1 match on English/code/French/chat prompts); tokenizer verified against AutoTokenizer.
• Conversion chain: FP8 → Q8_0 master (317.6 GB) → imatrix → per-tensor requants.
• Recipe A/B'd on held-out wikitext-2 before release (see PPL above); IQ4 split edition sanity-checked 4/4 on adversarial math/code prompts.