Qwythos-9B-Claude-Mythos-5-1M-NVFP4

NVFP4 (NVIDIA FP4, weight-only NVFP4A16) quantization of empero-ai/Qwythos-9B-Claude-Mythos-5-1M — a Claude-Mythos/Fable-trace reasoning fine-tune of Qwen3.5-9B (qwen3_5: a dense hybrid GatedDeltaNet (linear-attention) + full-attention model with a vision tower and a 1M-token YaRN context).

Variant: NVFP4A16 — 4-bit NVFP4 (FP4, E2M1 with FP8 per-block micro-scales + FP32 global scale, group size 16) weights; activations BF16. Built for NVIDIA Blackwell (e.g. Thor / RTX 50xx) FP4 tensor cores. Disk size: ~11.2 GB (vs ~18.8 GB BF16). The reduction is modest because the GatedDeltaNet linear_attn layers (24 of 32), the 248K-vocab embeddings + lm_head, and the vision tower are all kept in BF16; only the attention and MLP linears are FP4. Quantized by: sahilchachra Tooling: llm-compressor 0.12 model_free_ptq (data-free) → compressed-tensors

What is quantized

Quantized to NVFP4 (128 modules): self_attn.{q,k,v,o}_proj on the 8 full-attention layers + mlp.{gate,up,down}_proj on all 32 layers. Kept BF16: the GatedDeltaNet linear_attn layers, the vision tower, the MTP head, token embeddings, lm_head, norms.

Method

Data-free weight-only PTQ via llm-compressor's model_free_ptq — streams the safetensors from disk and applies NVFP4 block scaling directly to the weights (no calibration set, no full model load). Each quantized tensor stores a weight_packed (uint8-packed FP4), an FP8 (E4M3) per-block weight_scale, and an FP32 weight_global_scale.

Serving (vLLM)

from vllm import LLM, SamplingParams
llm = LLM(model="sahilchachra/Qwythos-9B-Claude-Mythos-5-1M-NVFP4",
          trust_remote_code=True)
print(llm.generate(["Explain a TCP SYN flood, briefly."],
                   SamplingParams(max_tokens=256))[0].outputs[0].text)

Best on Blackwell FP4 hardware. GatedDeltaNet kernels fall back to pure-PyTorch where unavailable. Load via vLLM (its own compressed-tensors loader).

Notes

Inherits the base model's uncensored behavior and 1M-token (YaRN) context.
The source repo omitted image/video *_processor_config.json; the standard Qwen3.5 (VL) processor configs are included here so the model loads in vLLM out of the box.
NVFP4A16 (weight-only) chosen for quality + broad loadability; activations stay BF16.
Smoke-tested (loads in vLLM + coherent generation); not a full quality benchmark.