ubergarm

/

DeepSeek-R1-0528-GGUF

like 47

Awesome! And glad to hear I wasn't alone. Out of curiosity, how much RAM do you have? I assume you have 12 channels?

I'm wondering if this issues has something to do with the new mla. Have you tried with -mla 2 and withouth -op 26,0,27,0,29,0?

I'm wondering if this issues has something to do with the new mla. Have you tried with -mla 2 and withouth -op 26,0,27,0,29,0?

I did try this, and it didn't make a difference. The only thing that made a difference was either using your Q4_K_R4 quants or disabling those tensors.

I always meant to test re-enabling one of the tensors at a time, so I just gave it a quick try. With -op 27,0,29,0 (re-enabling op 26), I get a little bit better performance. You may want to test the same @sousekd .

Awesome! And glad to hear I wasn't alone. Out of curiosity, how much RAM do you have? I assume you have 12 channels?

12 channels, DDR5 6400 on a single socket, 768 GB. I guess it is the reason for quite nice TG t/s.

I'm wondering if this issues has something to do with the new mla. Have you tried with -mla 2 and withouth -op 26,0,27,0,29,0?

PS>  .\bin\llama-sweep-bench.exe `
    --alias $ModelAlias `
    --model $ModelPath `
    --no-mmap `
    -mla 2 -fa -fmoe `
    -amb 512 -b 4096 -ub 2048 `
    -ctk f16 `
    -c 32768 `
    -ngl 63 `
    -ot "blk\.(3)\.ffn_.*=CUDA0" `
    -ot exps=CPU `
    --parallel 1 `
    --threads 32 `
    --threads-batch 32 `
    --warmup-batch

@anikifoss Same issue. I thought it is caused by Windows... or how I compile and build ik_llama, but I assume @Kebob is on Linux.
Anyway, just finished downloading your GGUF. I'll give it a ride tomorrow :).

I always meant to test re-enabling one of the tensors at a time, so I just gave it a quick try. With -op 27,0,29,0 (re-enabling op 26), I get a little bit better performance. You may want to test the same @sousekd .

Hmm, doesn't seem to work for me:

XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MUL_MAT to OFF
XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MUL_MAT_ID to OFF

main: n_kv_max = 32768, n_batch = 4096, n_ubatch = 2048, flash_attn = 1, n_gpu_layers = 63, n_threads = 32, n_
threads_batch = 32

Sanity check:

XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MUL_MAT to OFF
XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MUL_MAT_ID to OFF
XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MOE_FUSED_UP_GATE to OFF

main: n_kv_max = 32768, n_batch = 4096, n_ubatch = 2048, flash_attn = 1, n_gpu_layers = 63, n_threads = 32, n_
threads_batch = 32

@Kebob where did you got the idea BTW? I could not find anything about that when I tried :).

@Kebob where did you got the idea BTW? I could not find anything about that when I tried :).

It was just an idea I had. I'll need to do more testing as I originally tested it on https://huggingface.co/ubergarm/Qwen3-235B-A22B-GGUF. I just re-tested against the IQ4_KS_R4 quants in this repo and I see no difference.

It was just an idea I had.

I wish I have ideas like that! :)

Thanks for sharing all your tips for the 5090 club! I am wondering if it has something to do with the _r4 flavors I have been using or not and not so much the ks vs k quants, but haven't had a chance to dig into it more closely.

fwiw I'm currently uploading a new iq3_ks that is not _r4 that could be interesting for y'all to try out. I might make a larger and smaller version of it eventually too if there is interest.

https://huggingface.co/ubergarm/DeepSeek-TNG-R1T2-Chimera-GGUF

Hmm, doesn't seem to work for me:

Okay, taking back this one. It was late yesterday and I may have made a mistake. Also, it seems to me that the server (or ik_llama) has its moods, giving quite a different results from time to time. Anyway, in today's mood the results seems to be in favor of -op 27,0,29,0, especially on longer context, and not hit as hard on shorter context as it seemed when I posted the previous results.

PS>  .\bin\llama-sweep-bench.exe `
    --alias $ModelAlias `
    --model $ModelPath `
    --no-mmap `
    -mla 3 -fa -fmoe `
    -amb 512 -b 4096 -ub 2048 `
    -ctk f16 `
    -c 32768 `
    -ngl 63 `
    -op 26,0,27,0,29,0 `
    -ot "blk\.(3)\.ffn_.*=CUDA0" `
    -ot exps=CPU `
    --parallel 1 `
    --threads 32 `
    --threads-batch 32 `
    --warmup-batch

PS>  .\bin\llama-sweep-bench.exe `
    --alias $ModelAlias `
    --model $ModelPath `
    --no-mmap `
    -mla 3 -fa -fmoe `
    -amb 512 -b 4096 -ub 2048 `
    -ctk f16 `
    -c 32768 `
    -ngl 63 `
    -op 27,0,29,0 `
    -ot "blk\.(3)\.ffn_.*=CUDA0" `
    -ot exps=CPU `
    --parallel 1 `
    --threads 32 `
    --threads-batch 32 `
    --warmup-batch

fwiw I'm currently uploading a new iq3_ks that is not _r4 that could be interesting for y'all to try out. I might make a larger and smaller version of it eventually too if there is interest.

Oh that's great! I definitely have an interest in the larger version offering the best possible quality, even at the expense of some speed - the whole point of building this server :). The plan is to use smaller models where the speed is important, but have a model as smart and knowledgable as possible, even in obscure topics and languages, where and when needed. Thank you very much for all your work!

Sorry to resurrect the old thread, but I had a question:
Is -op 26,0,27,0,29,0 still useful, or is it a thing of the past?

@anikifoss The numbers (26, 27, 29) have changed. Based on @ikawrakow 's comments, it is probably still a thing for systems with slow PCI. See a related discussion here. I might do some tests later today with your DS-3.1-Terminus (downloading...).

Yes, the ops have changed. This is the current set of operations that can lead to large tensor offloads from CPU to GPU:

27:  MUL_MAT
28:  MUL_MAT_ID
30:  FUSED_UP_GATE
31:  MOE_FUSED_UP_GATE

Sorry about that. The change resulted from adding an op that was necessary for a model, and me not realizing that I should put the op at the end to not affect the -op command line argument, and putting the op at its more logical place instead. From this we got from (26,27,29) to (27,28,30). Then I added the fused ffn_up+ffn_gate operation, so it became (27,28,30,31)

@anikifoss So on my machine the -op 27,0,28,0,30,0,31,0 speeds up TG by 1-2 t/s but slows down PP by 20% (HQ4 Terminus).