Qwen3.5-122B-A10B-tq3a-tqTe-g64

Ternary (1.58-bit) expert TurboQuant quantization of Qwen/Qwen3.5-122B-A10B — 3-bit attention/dense, ternary {-c, 0, +c} experts packed as base-3 trits (group size 64) — produced with TurboQuant-MLX.

A 122B-parameter MoE compressed from ~240 GB (BF16) to 30.9 GB — a 7.8× reduction, and 43% smaller than the 3-bit build (54 GB) of the same model. Runs fully resident on 48 GB Macs (36 GB with a wired-limit bump), and streams its experts from SSD on machines as small as a 16 GB Mac mini.

Model Details

• Base model: Qwen/Qwen3.5-122B-A10B — Mixture-of-Experts (qwen3_5_moe, VLM; this build is used text-first)
• Architecture: 48 layers, hidden size 3072, 256 routed experts (top-8), MoE intermediate 1024, ~122B total / ~10B active params
• Quantization: TurboQuant (Hadamard rotation + Lloyd-Max codebook), group size 64
  • Attention / dense → 3-bit
  • Experts → ternary 1.58-bit (gate/up/down_proj of every routed expert): the data-free {-c, 0, +c} codebook, indices packed as genuine base-3 trits — 20 per uint32 = ~1.6 bpw
  • Routers → full precision (auto-skipped)
• Size: 30.90 GB (vs ~240 GB BF16; vs 54 GB for the tq3 sibling)

Why ternary works here

Ternary experts need redundancy to average out the coarse quantization, and this model has the best profile in the catalog: top-8 of 256 experts per token, quantized from a bf16 source. (For contrast: 32-expert models are word salad at sub-2-bit, and a 128-expert model quantized from an already-4-bit source degenerates too — expert count alone isn't the gate; it's experts × top-k routing × source precision.)

Note: this is a thinking-capable model. Use the chat template (the generator applies it automatically), and see the thinking-mode notes below.

Quality

Validated with a 6-probe stress suite on a 64 GB Apple Silicon Mac (fully resident, temp 0.7). All six probes pass:

Probe	Result
1500-word essay (Roman Empire history)	✅ 12K chars, coherent, no degeneration
Multi-step math (two-trains meeting problem)	✅ Correct meeting time and distance
Code generation (merge_intervals + unit tests)	✅ Correct function with assert-based tests
Needle-in-haystack recall (password in ~4K-word filler)	✅ Returned the password exactly
Strict format (numbered list, per-line word limits)	✅ Perfect list under --no-think (see note)
Repetition trap (long open-ended explanation)	✅ No repetition collapse

Thinking-mode note: like other low-bit thinking builds, the deliberation phase is the fragile part — it can overrun tight max_tokens caps on strict-format tasks. As of turboquant-mlx-full 0.12.3 the mitigations ship as defaults: this repo's generation_config.json carries a light repetition_penalty (1.05) which the CLI applies automatically for thinking sessions, a second </think> is masked (kills doubled answers), and --no-think gives instant, loop-immune answers — recommended for math, strict formats, and latency-sensitive use.

Running it

Requires turboquant-mlx-full ≥ 0.12.3.

pip install "turboquant-mlx-full>=0.12.3" mlx-lm

Fully resident (48 GB Mac; on 36 GB first run sudo sysctl -w iogpu.wired_limit_mb=34816):

python -m turboquant_mlx.generate \
    --model manjunathshiva/Qwen3.5-122B-A10B-tq3a-tqTe-g64 \
    --prompt "Explain why the sky is blue." --max-tokens 512

Measured on a 64 GB M4 Max (fully resident): ~13 tok/s decode, 32.5 GB peak.

Expert streaming (16–32 GB machines — pages only the router-selected experts from SSD):

python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/Qwen3.5-122B-A10B-tq3a-tqTe-g64 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512 --cache-budget-gb 8

Measured on a 16 GB M4 Mac mini (internal SSD, --cache-budget-gb 8): ~2.1 tok/s end-to-end, 11.3 GB peak RSS, 82% expert-cache hit rate. Adding --max-active-experts 3 reached ~2.4 tok/s with quality intact; pushing K lower gains little — the bottleneck is per-layer read latency across 48 MoE layers, not bandwidth. Set expectations accordingly: on 16 GB this model works, producing coherent full-quality answers, but a thinking-mode answer takes several minutes. Notes: --cache-budget-gb 8 is the ceiling on 16 GB (larger budgets exhaust RAM), and streaming this build reads ~43% fewer bytes per token than the 54 GB tq3 sibling. For an interactive 16 GB experience, use the 35B ternary sibling (fully resident, ~10 tok/s) — this 122B earns its keep fully resident on 36 GB+ Macs.

For an OpenAI-compatible server: pip install "turboquant-mlx-full[serve]>=0.12.3" then turboquant-serve --model manjunathshiva/Qwen3.5-122B-A10B-tq3a-tqTe-g64 --kv-bits 8 (add --cache-budget-gb 4 on small machines; --chat-template-args '{"enable_thinking": false}' to serve with thinking off).

Choosing a build

Build	Experts	Size	Fits resident	Decode (64 GB M4 Max, resident)	Notes
tq3a-tqTe-g64 (this)	ternary ~1.6 bpw	30.9 GB	36–64 GB, no sysctl needed on 48+	~13 tok/s	smallest; long-context headroom; fastest streaming on minis
tq3-g32	3-bit	54 GB	64 GB only, sysctl required	~25 tok/s (but Metal-OOMs past ~4–8K context)	fastest resident decode; recall-critical work

The trade in one line: ternary halves the decode speed but halves the footprint — base-3 trit decode costs more GPU ALU than 3-bit shift-and-mask. Pick tq3 for speed on a 64 GB machine at short context; pick this build for smaller Macs, zero setup, and room for long contexts.

How it was made

python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3.5-122B-A10B \
    --mlx-path Qwen3.5-122B-A10B-tq3a-tqTe-g64 \
    --bits 3 --group-size 64 --ternary-experts

Data-free (no calibration set): randomized Hadamard rotation → per-group RMS scaling → Lloyd-Max codebook (ternary for experts) → base-3 trit packing. The 3-entry codebook is self-describing on disk — loaders and Metal kernels switch to base-3 decode automatically.

Links

• TurboQuant-MLX on GitHub (Apache-2.0)
• PyPI: turboquant-mlx-full
• Sibling ternary builds: Qwen3.6-35B-A3B (9.4 GB — resident on a 16 GB mini) · Qwen3-235B-A22B (53 GB — resident on 64 GB)