The 9B MTP Q4_K_M model is looping its reasoning
Currently 9B Q4_K_M just loops its own thought paragraph by paragraph at complete random, it generates its reasoning for a good chunk then doing it again and again until it hits reasoning budget. I don't have that issue with normal Qwen 3.5 9B model. The sampling I used is the recommended one.
Hey @bakaemon
Can you provide more details like the harness you use, the tool you use to host the model aswell as the exact parameters?
A screenshot could also be helpful
It's llama.cpp server using vulkan back end, here is the command I am using:
llama-server \
--host 0.0.0.0 \
--port 11434 \
--mmap \
-c $((32*1024)) \
--flash-attn on \
--override-tensor "blk\.([1-9]|1[0-9]|2[0-9]|3[0-2])\..*=Vulkan0,.*=CPU" \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--reasoning on \
--reasoning-budget $((4*1024)) \
--reasoning-budget-message "OK, let's answer" \
-m /mnt/smt-08/HuggingFace/Models/Qwythos-9b-MTP-Q4_K_M.gguf \
--spec-type draft-mtp --spec-draft-n-max 2
The sampling followed strictly to your recommendation in the model card.
I am sorry for unable to provide the screenshot. The nature of the issue is quite random during the task . I could answer any question for provide further info. I will also try to produce a workable demonstration of the issue whenever possible.
If the model is looping, it's mostly your harness, the AI detects the loop but it doesnt have a way to exit the loop
If the model is looping, it's mostly your harness, the AI detects the loop but it doesnt have a way to exit the loop
then it does not explain why normal Qwen 3.5 9B does not having that issue, such is why I raised this to this thread.
one would say that explanation is that such finetunes are nothing more than hype
mine looped as well and it was not the harness lol
the finetune is a crap and downloads numbers just follows hype
Hey @bakaemon
From your command i see that you are missing a few blocks, i have adjusted it for you:
llama-server \
--host 0.0.0.0 \
--port 11434 \
--mmap \
-c $((32*1024)) \
--flash-attn on \
--override-tensor "blk\.([1-9]|1[0-9]|2[0-9]|3[0-2])\..*=Vulkan0,.*=CPU" \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--repeat-penalty 1.05 \
--n-predict 16384 \
--reasoning on \
-m /mnt/smt-08/HuggingFace/Models/Qwythos-9b-MTP-Q4_K_M.gguf \
--spec-type draft-mtp --spec-draft-n-max 2
You were missing the sampling recommendations and your reasoning budget was way to low for this model as its quiet verbose.
Also: Careful with kv cache quantization on qwen3.5 architecture. Small models were very sensitive to it in the originals as well.
Best for q3.5 9B: No kv cache quantization, stick exactly to temp, top_k etc.
Hey @bakaemon
From your command i see that you are missing a few blocks, i have adjusted it for you:
llama-server \ --host 0.0.0.0 \ --port 11434 \ --mmap \ -c $((32*1024)) \ --flash-attn on \ --override-tensor "blk\.([1-9]|1[0-9]|2[0-9]|3[0-2])\..*=Vulkan0,.*=CPU" \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --repeat-penalty 1.05 \ --n-predict 16384 \ --reasoning on \ -m /mnt/smt-08/HuggingFace/Models/Qwythos-9b-MTP-Q4_K_M.gguf \ --spec-type draft-mtp --spec-draft-n-max 2You were missing the sampling recommendations and your reasoning budget was way to low for this model as its quiet verbose.
Also: Careful with kv cache quantization on qwen3.5 architecture. Small models were very sensitive to it in the originals as well.
Best for q3.5 9B: No kv cache quantization, stick exactly to temp, top_k etc.
thank you for the suggestions. I applied the samplings from the client side to the API. Though currently I do not know how to replicate the issue as it happens randomly, and more likely during a heavy logical task like autonomous problem solving inside a VM. I will try as per suggestion setting sampling from server side.