Will crash EVERY time when the context is >240.000
No matter what quantization, or who quantized it, or what version of llama.cpp, it will crash every single time. No other model (and I use many) causes this. Really disapointing.
Hi @Nerdsking , do you mean 240K context length? Would you elaborate "crash"? Do you mean llama.cpp process crashes, or the model outputs non-sense tokens (i.e., the model "crashes")
Yep, 240K context length... The PC will crash, and boot. This is the only model where such a thing happens. I tried a lot of llama.cpp versions (compiled versions and "releases" versions for download), many types of quantizations from unsloth, and many other teams. The result is the same. I use lama-server in windows 11, then I send a group of files to analysis, and after a while, the system crash.... And I use mimo-2.5, Qwen 3.6, Gemma, Minimax 2.7. All of them run just fine. But stepfun model doesnt work. Not sure why. I gave up trying.
Hardware: Windows 11, in Ryzen 3700x with 128GB 3600Mhz RAM, RTX 2080ti 22G (modded), llama.cpp + Cuda 13.1. --- Qwen3.5-122B-A10B-UD-Q8_K_XL runs at about 9 tk/s. And MiMo-V2.5-IQ4_XS at about 4 tk/s with a context of 250 K.
@Nerdsking Weird.. Does shorter context length (e.g., 32K or even 4K) work? By "work" I mean your PC doesn't crash and the model can generate tokens. If so, it's related to memory management. Though I still don't get why llama.cpp does not quit gracefully while reporting OOM.
@apohelios Did you encounter similar problems when tested it on PC?
Shorter ones would not crash the pc. I didn't tested exactly at what size it starts crashing my system. I tried with 250k, then I reduced to 240k. And then I tried all possible source of stepfun. The fact is that ALL the other models I use behave normally, even working the whole day in bigger files, whithout problems. To crash the system, thats new to me, never saw this. Anyway, I gave up. Good luck finding the cause.