Lower quants?
Do you plan to release quants, which will fit to 128Gb DDR + 24Gb VRAM?
They're coming yes, not verified until we announce them
Awesome! Hoping I can run on a 128gb device.
They're coming yes, not verified until we announce them
Когда сделаете 1бит пж быстра
They're coming yes, not verified until we announce them
Can you also do a quant for 64Gb DDR + 24 Gb VRAM? The size should be around 78-80gb
They're coming yes, not verified until we announce them
Can you also do a quant for 64Gb DDR + 24 Gb VRAM? The size should be around 78-80gb
I think it is impossible. besides model, kv cache maybe around 10gb
They're here!!
They're here!!
можете сделать квантизацию чтобы в 8гигабайт поместилось просто нету больше памяти
We found DeepSeek-V4 issues in llama.cpp that caused gibberish after the 2nd turn. The cause was broken prompt caching. To run correctly, please use the latest llama.cpp version.
We also improved the DeepSeek-V4 chat jinja template, and tested over 4000 conversations to be equivalent with the official baseline.
Guide: https://unsloth.ai/docs/models/deepseek-v4
GGUF: https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF
You can run DeepSeek-V4-Flash with all our fixes and Thinking toggles via Unsloth Studio:
llama.cpp added DeepSeek V4 support in #24162 - we noticed that when using any GGUF from any provider, multi turn conversations would not function well when compared to DS4's Hugging Face baseline. llama.cpp uses --ctx-checkpoints N which allowed it to do prefix caching to save inference costs. Instead of re-processing every prompt again on the 2nd, nth ask, we can use KV caching. However we found DS4 needed --ctx-checkpoints 0 or else you will get gibberish. Please use the latest version of llama.cpp to get fixes.
| Engine | Score | Calculation | Tool selection | Parallel Tools | Multi Turn tools | Nested tools |
|---|---|---|---|---|---|---|
| Official code | 15/15 | 3 | 3 | 3 | 3 | 3 |
| Any provider | 4/15 | 1 | 2 | 0 | 0 | 1 |
| After our fix | 15/15 | 3 | 2 | 3 | 3 | 3 |
Thanks guys and feel free to support our Tweet, Linkedin post or Reddit post
CC: @Kelheor @dziliak @EurusWine @amane2 @LevLv @Kosh69 @eepos @Schnuckade @ParadigmComplex
Running DS 4 Flash Q3_K_XL on a Ryzen 5950x, 128 GB RAM, and 9060 XT 16 GB rocm.
Both the CPU and GPU are running on base clocks in order to achieve energy economy.
Context 96K (2^16 + 2^15), options: --fit on, --cpu-moe.
Using these options, the model perfectly fits into RAM and VRAM (swap is not used).
Performance: input 45-24 t/s, output 4-3 t/s.
It is fine for my workload.
llama-b9936-bin-win-cuda-13.3-x64, UD-IQ3_XXS, 5950x, 128Gb DDR4, RTX 4090
--n-cpu-moe 42 --no-mmap -b 2048 -ub 1024
input 120-140 t/s, output 8-9.5 t/s
--ctx-size 80000 (with such a slow input speed larger context and agentic coding will be too slow anyway).
UPD: it's possible to increase context to 100000 with --n-cpu-moe 44, the output speed will be reduced by 0.5 t/s.
However, I didn't find a way to further increase context size without speed loss, since even --n-cpu-moe 50 won't be enough to fit 128k context, for example, due to limited VRAM.
-ub 1024 - I also don't want to lower that value, since input speed will be much slower in that case.