Writer/qwen3.6-35b-a3b-cigna-palmyra-asft-fp8-blockwise

qwen3.6-35b-a3b-cigna-palmyra-asft-fp8

FP8 (W8A8-dynamic) quantization of the Cigna–Palmyra ASFT fine-tune of Qwen/Qwen3.6-35B-A3B, produced by (1) merging the LoRA adapter Writer/qwen3.6-35b-a3b-cigna-palmyra-asft (subfolder asftnr_lr1e-4_ep2_kl0.03) into the bf16 base, then (2) quantizing the merged model to FP8 with llm-compressor.

The model is a multimodal MoE (Qwen3_5MoeForConditionalGeneration): 256 routed experts (top-8), a shared expert, hybrid full/linear attention, a vision tower, and an MTP (multi-token-prediction) head. It serves out-of-the-box in vLLM.

• Weights: FP8 E4M3 (per-channel) for the language-model Linear layers and the routed experts; bf16 for the vision tower, MoE router gates, shared-expert gates, linear-attention projections, lm_head, embeddings, norms, and the MTP head.
• Activations: FP8 dynamic, per-token.
• On disk: ~37 GB (vs. ~66 GB for the merged bf16 model).

---

How this model was produced (reproducible steps)

0. Environment

The exact stack used (an isolated uv venv, Python 3.12):

package	version
torch	2.12.0 (cu130)
transformers	5.10.1
llmcompressor	0.12.0
compressed-tensors	0.17.1
peft	0.19.1
accelerate	1.14.0

Note: transformers >= 5.10 is required — the qwen3_5_moe architecture is not recognized by older releases (and it ships no remote code, so trust_remote_code cannot substitute). Installing llmcompressor pins transformers to 5.10.1, which still supports the architecture.

uv venv --python 3.12 .venv
uv pip install torch==2.12.0 transformers peft accelerate safetensors llmcompressor

1. Merge the LoRA adapter — including the fused routed experts

This is the subtle step. The base stores its 256 routed experts fused as 3-D parameters:

Qwen3_5MoeExperts.gate_up_proj  [E=256, 2*I=1024, H=2048]   # gate = rows 0:I, up = rows I:2I
Qwen3_5MoeExperts.down_proj     [E=256, H=2048,   I=512]

but the adapter stores routed-expert LoRA per-expert and unfused (...mlp.experts.{e}.{gate,up,down}_proj.lora_{A,B} — 61,440 tensors, ~99% of the adapter). A plain PEFT target_modules merge on the fused base matches none of these expert modules and silently drops the entire routed-expert fine-tune (keeping only the 350 attention / linear-attn / shared-expert LoRAs). The model's learned behavior (including its "Palmyra / Writer" identity) lives in those routed experts, so they must be merged.

The delta is added directly into the correct slice of each fused tensor — mathematically identical to unfuse → PEFT-merge → re-fuse, but with no round-trip and no layout risk. Split order (gate first) is confirmed from modeling_qwen3_5_moe.py: gate, up = linear(x, gate_up_proj[e]).chunk(2, dim=-1).

# scaling = lora_alpha / r = 32 / 16 = 2.0  (uniform: no rslora/dora/rank_pattern)
# for each adapter LoRA pair (A:[r,in], B:[out,r]):
delta = 2.0 * (B @ A)                       # [out, in], computed in fp32
# routed expert e:
#   gate: gate_up_proj[e, 0:I]  += delta
#   up:   gate_up_proj[e, I:2I] += delta
#   down: down_proj[e]          += delta
# everything else (self_attn / linear_attn / shared_expert): weight += delta
# fp32 accumulate, cast back to bf16

The load class must be AutoModelForImageTextToText (= Qwen3_5MoeForConditionalGeneration) — not AutoModelForCausalLM, which exposes the text tower as model.layers.* (no language_model. prefix, no vision) and fails to match the adapter.

Validation performed during the merge: all 31,030 LoRA pairs resolve (30,720 expert + 310 linear), a numeric spot-check of a merged expert slice (max|applied − expected| ≈ 1.5e-4, bf16 rounding), and every parameter is bf16.

Result: a 66 GB bf16 merged checkpoint with the full fine-tune baked in.

The base model's MTP head is dropped on load by the ForConditionalGeneration class, so the merged bf16 checkpoint has no MTP tensors. The adapter does not touch MTP; the head is restored (bf16) from the base during quantization (below).

2. FP8 quantization with llm-compressor

FP8 for this Qwen3.5-MoE-VL family is data-free (no calibration dataset) — this is the officially supported path (FP8_BLOCK). llm-compressor's load_context unfuses the 256 experts into per-expert Linears so they are actually quantized, then re-fuses on save for vLLM.

See https://github.com/WriterInternal/nlp.cigna-lora-infra-evals/tree/main/quantization-v2

Result: ~37 GB — 30,880 FP8 weight tensors (+ scales), the rest bf16, plus a separate model_mtp.safetensors (19 bf16 MTP tensors).

---

Serving with vLLM

vLLM auto-detects the compressed-tensors / float-quantized config — no --quantization flag needed.

vllm serve qwen3.6-35b-a3b-cigna-palmyra-asft-fp8 \
    --served-model-name palmyra-cigna-fp8 \
    --tensor-parallel-size 4 \
    --max-model-len 65536 \
    --reasoning-parser qwen3 \
    --trust-remote-code

Disable the thinking/reasoning trace per request with Qwen's template kwarg:

{"model": "palmyra-cigna-fp8",
 "messages": [{"role": "user", "content": "Who are you?"}],
 "chat_template_kwargs": {"enable_thinking": false}}

Requires FP8-capable hardware (H100 / H200). Tested on 4× H100.

---

Provenance & licensing

Derived from Qwen/Qwen3.6-35B-A3B and the Writer/qwen3.6-35b-a3b-cigna-palmyra-asft LoRA adapter (subfolder asftnr_lr1e-4_ep2_kl0.03). Use is subject to the license terms of the base model and the adapter. The stock Qwen chat template is used unmodified.