bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF

GGUF / llama.cpp quantizations of bottlecapai/ThinkingCap-Qwen3.6-27B — capability of Qwen3.6-27B with 50% less thinking tokens on average, achieved by finetuning Qwen3.6-27B (Qwen Team, 2026) with online reinforcement learning while preserving the original answer quality and style.

➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.6-27B.

About GGUF and quantization

GGUF is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (Ollama, LM Studio, …). The quantized variants below store weights at reduced precision — e.g. ≈4.7 bits per weight for Q4_K_M instead of the 16-bit f16 source — cutting download size and memory severalfold at a small, measured quality cost.

Files

File	Quant	Size
ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf	Q4_K_M	15.7 GB
ThinkingCap-Qwen3.6-27B-Q8_0.gguf	Q8_0	27.1 GB
ThinkingCap-Qwen3.6-27B-f16.gguf	f16	50.9 GB
mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf	mmproj (vision)	0.9 GB

f16 is the unquantized source; Q8_0 is near-lossless; Q4_K_M is the recommended size/quality balance for most local setups.

Usage (llama.cpp)

# pull a specific quant straight from the Hub and chat
llama-cli -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M -p "Hi"

# or download one file and run it
huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf --local-dir .
llama-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf -p "Hi"

Speculative decoding (MTP)

llama.cpp can run MTP (multi-token-prediction) self-speculative decoding on these GGUFs for a decode speed-up — no separate draft model needed. Add --spec-type draft-mtp when serving:

llama-server -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M --spec-type draft-mtp

Set the draft length with --spec-draft-n-max (e.g. 4). Requires a recent llama.cpp build with MTP support.

Vision (image input)

ThinkingCap is a vision-language model. Image input needs the multimodal projector mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf (in this repo) loaded alongside a text GGUF — the single f16 mmproj pairs with any of the quants above.

• LM Studio / Jan / Ollama, …: download the mmproj-*.gguf from this repo; LM Studio auto-detects it and enables the image (🖼️) button.
• llama.cpp CLI:

huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF \
  ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --local-dir .
llama-mtmd-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --image photo.jpg -p "Describe this image."

• llama-server: add --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf to expose an OpenAI-compatible vision endpoint.

Expected performance

From our internal serving-validation harness (llama.cpp, single-stream, temperature 0) on a fast N=100/dataset subset of MMLU-Pro (reasoning) and RealWorldQA (vision) — a quick quant-parity + decode-speed check, not the headline accuracy evals (for the multi-seed, statistically-tested results see the main model card).

Our three quants (f16/Q8_0/Q4_K_M) stay within subset noise of f16 on accuracy, and MTP self-speculative decoding (--spec-type draft-mtp, n=4) accepts ≈3.75 tokens per verify step — a ≈1.4–1.7× per-token decode speed-up on top of the finetune's ≈50% token savings. The two bolded rows are our picks: Q8_0 + MTP is the fastest per task on our hardware (it out-decodes Q4_K_M here) and near-lossless; Q4_K_M + MTP is the smaller size/quality balance for tighter memory budgets. For reference we also list unsloth's Dynamic GGUFs of the base model (UD-*): same llama.cpp path, but base-model quants — so they match base accuracy and reason ≈2× longer (none of the finetune's token savings).

median tokens = median completion length; task s = median tokens ÷ single-stream tok/s (real per-request time); speedup is vs the unquantized base model (bf16 GGUF) in standard decoding — same llama.cpp path as every row, so the comparison is apples-to-apples.

MMLU-Pro (reasoning)

config	acc	median tokens	tok/s	task s	speedup	accept_len (n=4)
Qwen3.6-27B base (bf16 GGUF) · standard	0.83	1999	50.4	39.6	1.00×	—
f16 · standard	0.89	884	50.4	17.5	2.26×	—
f16 · MTP	0.88	870	86.7	10.0	3.96×	3.78
Q8_0 · standard	0.88	890	57.2	15.6	2.54×	—
Q8_0 · MTP	0.86	856	99.4	8.6	4.60×	3.77
Q4_K_M · standard	0.86	814	61.8	13.2	3.00×	—
Q4_K_M · MTP	0.85	848	89.2	9.5	4.17×	3.74
unsloth UD-Q8_K_XL (base) · standard	0.85	1896	54.5	34.8	1.14×	—
unsloth UD-Q8_K_XL (base) · MTP	0.86	1925	98.2	19.6	2.02×	3.74
unsloth UD-Q4_K_XL (base) · standard	0.84	1976	62.1	31.8	1.25×	—
unsloth UD-Q4_K_XL (base) · MTP	0.83	1928	87.1	22.1	1.79×	3.72

RealWorldQA (vision)

config	acc	median tokens	tok/s	task s	speedup	accept_len (n=4)
Qwen3.6-27B base (bf16 GGUF) · standard	0.66	612	50.4	12.1	1.00×	—
f16 · standard	0.79	271	50.4	5.4	2.24×	—
f16 · MTP	0.79	271	86.7	3.1	3.90×	3.78
Q8_0 · standard	0.79	270	57.2	4.7	2.57×	—
Q8_0 · MTP	0.78	273	99.4	2.7	4.48×	3.77
Q4_K_M · standard	0.78	283	61.8	4.6	2.63×	—
Q4_K_M · MTP	0.78	274	89.2	3.1	3.90×	3.74
unsloth UD-Q8_K_XL (base) · standard	0.68	530	54.5	9.7	1.25×	—
unsloth UD-Q8_K_XL (base) · MTP	0.69	550	98.2	5.6	2.16×	3.74
unsloth UD-Q4_K_XL (base) · standard	0.65	655	62.1	10.5	1.15×	—
unsloth UD-Q4_K_XL (base) · MTP	0.70	564	87.1	6.5	1.86×	3.72