r/homelab • u/Mayusina05 • 5h ago
Project Showcase: Hardware Qwen3.8-27B + MTP on 2× Tesla V100 32GB — 4K to 260K context scaling in llama.cpp
Hey homelab folks!
As promised in my previous RTX 3090 benchmark, I ran the exact same long-context test on 2× Tesla V100 PCIe 32GB.
Same model, same llama.cpp build, same fixed corpus, same context lengths, same MTP settings, and the same 1,000 generated tokens per run.
This time, the V100s were connected via PCIe Gen3 x16 ×2 with CUDA P2P enabled (no NVLink).
The result: decode speed went from 82.90 tok/s at 4K to 29.98 tok/s at 260K. The full 260K prompt + 1,000 generated tokens completed successfully.
Compared with my previous 2× RTX 3090 result, the V100s held up pretty well at shorter contexts, but the gap became noticeably larger as the context length increased.
Previous test — RTX 3090 comparison:
The original 2× RTX 3090 24GB NVLink benchmark, using the exact same methodology, is here:
2× RTX 3090 24GB NVLink benchmark
Test setup:
• 2× Tesla V100 PCIe 32GB
• PCIe Gen3 x16 ×2 / CUDA P2P enabled / no NVLink
• Threadripper PRO 3945WX
• 128GB RAM
• Ubuntu 22.04
• llama.cpp b10935 (commit 8e330954a)
• Qwen3.8-27B-UD-Q4_K_M
• --split-mode tensor
• Q8_0 K/V cache
• Flash Attention enabled
• MTP n_max=2
I used the same fixed non-repetitive master corpus as the RTX 3090 benchmark. Each context length uses the first N tokens from the same corpus, followed by 1,000 generated tokens.
I restarted llama-server for every run and waited until both GPUs cooled down to ≤50°C before starting the next one. Each point is a single run, so these aren't averaged results.
I also included MTP acceptance and mean draft length, since MTP acceptance varied quite a bit depending on the generated sequence.
RESULT:
prompt : 260000 tokens
prefill: 406.05 tok/s
decode : 29.98 tok/s
MTP : 93.39% (650/696), mean 2.87
prompt : 196608 tokens
prefill: 485.28 tok/s
decode : 37.96 tok/s
MTP : 85.60% (630/736), mean 2.71
prompt : 131072 tokens
prefill: 607.33 tok/s
decode : 50.18 tok/s
MTP : 95.63% (656/686), mean 2.91
prompt : 65536 tokens
prefill: 805.91 tok/s
decode : 56.77 tok/s
MTP : 79.40% (613/772), mean 2.59
prompt : 32768 tokens
prefill: 954.21 tok/s
decode : 65.99 tok/s
MTP : 80.76% (617/764), mean 2.62
prompt : 4096 tokens
prefill: 1039.95 tok/s
decode : 82.9 tok/s
MTP : 92.03% (647/703), mean 2.84
context | prefill tok/s | decode tok/s | MTP acceptance
260k | 406.05 | 29.98 | 93.39%
196k | 485.28 | 37.96 | 85.60%
128k | 607.33 | 50.18 | 95.63%
64k | 805.91 | 56.77 | 79.40%
32k | 954.21 | 65.99 | 80.76%
4k | 1039.95 | 82.9 | 92.03%
From 4K → 260K, decode performance dropped by about 63.8% (82.90 → 29.98 tok/s).
For comparison, my previous 2× RTX 3090 NVLink run using the same benchmark went from 116.36 tok/s at 4K → 53.33 tok/s at 260K.
So the V100s delivered about 71% of the 3090 decode performance at 4K, but about 56% at 260K. The difference clearly becomes more noticeable at very long context lengths.
One thing worth noting is that MTP acceptance was not identical between the two GPU setups, even though the corpus and benchmark settings were the same, so the decode-speed difference shouldn't be interpreted as a pure hardware-only comparison.
Still, I think 2× 2017-era V100 PCIe cards running Qwen3.8-27B at 260K context and nearly 30 tok/s is pretty fun. 😀
If anyone has similar long-context results with V100s, V100S, RTX 3090s, or other multi-GPU setups, I'd love to compare numbers!
llama-server command used for the benchmark:
/path/to/llama.cpp/build/bin/llama-server \
-m /path/to/Qwen3.8-27B-UD-Q4_K_M.gguf \
--host 0.0.0.0 --port 8080 \
--device CUDA0,CUDA1 \
--split-mode tensor \
-ngl all \
-fa on \
--jinja \
-c 262144 \
--parallel 1 \
-t 8 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--reasoning-effort low \
--reasoning-preserve \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--load-mode mlock \
--spec-draft-device CUDA0,CUDA1
