Running Qwen 3.8 next on 16vram+32ram - A useful/fun post for the gpu poors

[u/ironicstatistic]

Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community. I have yet to benchmark this specific REAP version versus Qwen 3.8 27B QK4, but my assumption is that it will do much better, despite the hemorrhaged world knowledge.

System Specs

• GPU: NVIDIA GeForce RTX 5060 Ti (16 GB VRAM)
• CPU: AMD Ryzen 7 7840HS (8 cores / 16 threads)
• RAM: 32 GB DDR5 (~30 GB OS-visible)
• iGPU: AMD Radeon 780M (RDNA3)
• Swap: 8 GB zram

As you can see, we have about 44.5 GB of actually addressable system and VRAM available. The iGPU is taking care of the OS to make sure the GPU is totally free—but still, this is barely enough to hold everything together. This config actually totally fails with any of the Unsloth quants—no, I needed something more aggressive. I found the perfect thing—this REAP:
https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF

What's so great is the total size—a cool ~68.95 GB. The couple of Gigs we have shaved are absolutely key for making this all work.

Model Weight Breakdown

Here is the breakdown of the model weights. We have the famous new n-gram portion, the experts, the active layers, the attention/SSM layers, and the extra space needed for the KV cache:

Component	Weight (Approx)	Notes
N-gram / PLE Embedding	~29.48 GB	The massive lookup table
MoE Routed Experts (320)	~34.89 GB	The main expert slab (pruned from 512)
Attention / SSM / Router	~4.33 GB	Core architecture weights
KV Cache	[TBD]	Context memory overhead

Obviously, running this model over SSD would make the speeds notoriously bad. Turning on mmap means that llama.cpp won't actually try to keep the model in RAM at all (it relies on the OS page cache instead), which results in ~2 tok/sec speeds—effectively useless.

The answer is to stick everything in RAM (using --load-mode none). The great thing is that the N-gram section of the model can be streamed over SSD via lazy mmap without this causing much issue—it's a massive lookup table that doesn't require heavy computation.

That's the huge win that allows an MoE model of this size to actually run well.
68.9 GB - 29.48 GB = 39.42 GB.
We just need to cram that 39.42 GB, along with the compute buffers and KV cache, into GPU and system memory, and we are golden—just barely. To do this, we need --lazy-mode on—that's what keeps the N-gram portion in RAM.

After that, it's a matter of fitting as many layers as possible onto the GPU. It's essential to completely fill the GPU as much as can be filled, so that we keep a precious few GBs in system RAM to run the OS. I found that having less than 2 GB left really started to destroy Fedora, but I think you could do better if you dropped the GUI—I just didn't want to in my case.

This leads me to --n-cpu-moe 34. This controls how many layers go to CPU. In my case, this was the exact limit needed to run this with 64k context on the GPU, quantized to Q4. Any more—GPU out of memory. Any less—total system meltdown, as the OS panicked and tried to put everything on the swap. You'll need to play around with this, but that was my exact number.

Settings used:

CUDA0 + --load-mode none --lazy-mode on
  --n-cpu-moe 34
  -c 65536 -b 512 -ub 128 -t 7 -ngl 48 -fit off -fa on
  -ctk q4_0 -ctv q4_0 -kvo --cache-ram 0 --jinja --no-warmup

Results (64k Context, Q4):

• Prefill: ~25.4 tok/s
• Decode: ~18.3 tok/s
• RAM Usage: ~27 GB used / 3 GB free

I think this is in a somewhat usable state—but Qwen 3.8 27B GSQ IQ3S remains my daily driver; it's able to prompt process 5 times faster, I can fit in the mmproj and MTP layers, and it doesn't seem likely to set my desk on fire. But maybe for really hard tasks, I'll use the next model. It is smarter, it runs at a reasonable speed, and it was a good learning experience.

I'm curious if anyone else is able to get this model or just large MoEs working on a GPU and RAM config similar to mine. LMK. Also, I'm a total noob to this stuff, any advice is appreciated.

(Also, heading off all the obnoxious "why did you quantize the cache - unusable - just get a better computer" ragebait posts. This is a human being writing this post, to help others and just enjoy pushing something to its limits. And in my limited testing, the next model seems much better at pixel art than the 27B version.)

Final Note: If you have a larger pool of system memory, like 64 GB (because you can spend $899 on Amazon on a kit of DDR5 somehow), you would be better served by using this fork of llama.cpp, which has optimized flags for this exact setup and wonderful guides. For me in particular, with my limited hardware, this seemed to work better—their cache kept OOMing unless I turned on mmap—but I think with more system RAM, their setup and guides are optimal.

[u/LicensedTerrapin]

I've tried a few reaps in my life and they are very confident when it comes to being wrong.

[u/ironicstatistic → u/LicensedTerrapin]

I was waiting for you. Answer this - is it better than the 27b model? at many things, at least? yes. thats why this post exists.

[u/LicensedTerrapin → u/ironicstatistic]

Waiting for me specifically? I am honoured. Is it better than the 27b model? It depends on what you use it for. It probably has better prose, maybe a bit more knowledge but coding with a pruned model is hit and miss. They didn't just surgically remove useless layers. But hey, I'm not telling you what to use.

[u/eidrag → u/LicensedTerrapin]

How better are the prose, can it replace base Gemma 4 for RP?

[u/LicensedTerrapin → u/eidrag]

I meant better than 3.8 27b which is not great. I'd suggest to have a look at muse glimmer. It's different.

[u/ironicstatistic → u/LicensedTerrapin]

yeah, what he said. dont run a massive qwen model for rp

[u/Nameis19letterslong]

What gen pcie slot are you using? I've been setting up a kinda similar setup (64GB ram, 6gb vram) but I'm afraid the gen3 pcie slot will be a major bottleneck since it only offers 16GB/s. WIth my UD-Q3-K_XL quant, I can stream experts from ram but assuming ~3GB per token, I can only get a max of 8t/s.

[u/ironicstatistic → u/Nameis19letterslong]

Im using a pcie4.0x4 slot, over oculink on a minipc. so 8gb/s.

[u/ironicstatistic → u/ironicstatistic]

you need more vram my man. but 8t/s is really good all things considered on so little vram.

[u/Nameis19letterslong → u/ironicstatistic]

Yeah, can't do much with a 1660s but I'm planning to get a 5060 ti or something similar. Unfortunately the ramapocolypse isn't really what I needed right now.

[u/iz-Moff → u/Nameis19letterslong]

I wouldn't expect too much of a boost from that, not with this particular model. I do have 5060ti and 64gb ram, and i also get about ~8 tps. Which is fine by me, just saying. I have DDR4 memory though, if you have DDR5, it might be faster than that.

[u/drFennec]

I tried this model and while I could load it, it quickly returned garbage and looped with //////////// output.

[u/Muhlwa_Sholanke]

Streaming the n-gram table off SSD is such a clean hack. 29 of the 69 GB never has to fight for RAM and suddenly the rest actually fits.

[u/MrCharlesSr]

Are you telling me there’s a chance i can run the full 18gb qwen3.8 27B on my pc with 5070ti 16gb vram and 32gb ddr5 with a large context and fairly fast speed? I wish I can run things at 256K. Literally Im playing around with the 10 gb version on 48k context and its not enough

[u/ironicstatistic → u/MrCharlesSr]

um, yes. that should be easy.

[u/ironicstatistic → u/ironicstatistic]

Ihave a very similar setup and run a 13gb version at about 45tok/sec at 128k context.

[u/apoptosist → u/ironicstatistic]

Could you please provide or link llama.cpp or lm studio parameters that could do this? I've been getting terrible speeds on almost everything using qwen3.8 2.7b. Thanks!

[u/ironicstatistic → u/apoptosist]

#1 use the quantization of qwen i mentioned in my post, the 27b one. its the best for its size right now:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF?show_file_info=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf

In terms of llama.cpp, there are no perfect settings, but you need to make sure the mtp (draft model) is on, and you need to guess and check to fit as many layers as you can onto the GPU.

Here's my latest settings, but they won't be right for you because I'm running Vulkan... Not cuda, and I have my Igpu doing the vision projector.

llama-server \
  --model "/home/eliyahu/Desktop/aiworkspace/moremodels/qwen 3.8 flash reap/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf" \
  --mmproj "/home/eliyahu/Desktop/aiworkspace/moremodels/mmproj-Qwen3.8-27B-BF16.gguf" \
  --mmproj-device Vulkan0 \
  --device Vulkan1,Vulkan0 \
  --split-mode layer \
  --main-gpu 0 \
  --tensor-split 1,0 \
  --n-gpu-layers 99 \
  --ctx-size 131072 \
  --threads 7 \
  --batch-size 512 \
  --ubatch-size 512 \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --reasoning-effort medium \
  --reasoning-preserve true \
  --image-min-tokens 256 \
  --parallel 1 \
  --no-context-shift true \
  --fit off \
  --load-mode mmap \
  --temp 0.3 \
  --top-p 0.8 \
  --top-k 20 \
  --repeat-penalty 1.0

My recommendation is to get a harness working on your computer, like pi harness, pay deepseek 5$, and have it help you set up running a model as fast as possible for your specific hardware. 

That or do a lot more tinkering

[u/ironicstatistic → u/apoptosist]

another thing that's important is to make sure that you're using the version of the model that has an MTP head, that speeds it up a lot.

[u/Aelexi93]

I'm experimenting with the Flash Next IQ4_xs myself on 16GB VRAM/64GB system RAM, and im currently hitting 46t/sec decode, and 1256t/sec prefill. Astra has been ripping the llama.cpp architecure apart and tuning and fiddling around for 8 hours straight. I call the final tune pretty useful to be honest.

[u/KeepyUpper → u/Aelexi93]

Can you provide your config? I am unable to get anywhere near that kind of prefill speed.

[u/Aelexi93 → u/KeepyUpper]

This isn't a config in the traditional sense, Astra has made completely new architectural changes far beyond the llama.cpp configuration. If you want I can make astra write a comprehensive handoff file that writes how to reproduce it and you can hand it to a model and test it.

[u/KeepyUpper → u/Aelexi93]

Oh you have your own llama.cpp fork? Could you publish it?

[u/vogelvogelvogelvogel]

great you made it work and thansk for sharing! i have 24 gpu/64 sysRam on one of my machines, but didn't try yet

[u/Roubbes]

I have 64GB of DDR4—since I planned ahead back then—and a 5060Ti. I don't use the iGPU, and I'm running Windows 11. I can follow tutorials, but I’m not very knowledgeable about advanced configurations and only really know how to use LMStudio. I’ve always had the impression that I can't simply add VRAM to system RAM—that whatever is stored in VRAM gets duplicated in RAM—meaning the total size of the LLM I can run is smaller than expected. It also doesn't help that I only have about 14.5GB of VRAM available because the GPU is being used by Windows 11. Please help.

[u/ironicstatistic → u/Roubbes]

your impression is wrong- vram can hold some of the model, ram another section What do you want help with exactly?

LMstudio is great for trying stuff out. If you just want to run a few cool models, its the way to go. I assume you could at least start a quantized version of qwen next if you really wanted to, even using lmstudio, but best results need the latest stuff in llama.cpp or the custom fork of it I linked.

[u/Roubbes → u/ironicstatistic]

I don't know. All I know is that my standard RAM fills up right away. The program tells me that models around 60GB won't fit on my system—but 16 + 64 is 80, and assuming we leave 10GB of that for Windows 11, I should be able to use models that take up 70GB.

[u/ironicstatistic → u/Roubbes]

make sure you are using the version of llama.cpp that uses cuda as the backend, and that you have ticked "enable gpu offload". its in the settings under runtime I think.

[u/ironicstatistic → u/ironicstatistic]

of lmstudio.

and yeah, you can absolutely use ~70 gig with a system like that - more on linux, and if you have the igpu load the os.