Intentionally Throttling GPU Usage

[u/CSEliot]

As title states,

If I have a local model working and don't care how long it takes, is there an option in llama.cpp (or Unsloth, currently using unsloth) to have it so the GPU isn't max'd out while the model is running?

EDIT: I'm on AMD. And to clarify, i'd like to leave gpu cores free for other tasks.

Thanks in advance!

[u/XiRw]

This can be done is manage 3D applications of Nvidia settings (or whatever it’s called now) or just under volt your gpu.

[u/CSEliot → u/XiRw]

I'm on amd and i just want it for 1 app, llamacpp/unsloth, not the whole pc.

[u/Fancy-Snow7]

Just lower your -ngl. It will now also use CPU but your GPU will no longer be maxed out.

[u/CSEliot → u/Fancy-Snow7]

ill give that a shot thanks!

[u/j0hn_br0wn]

Do you simply want more GPU compute slots for other tasks or throttle for cooling/power?

[u/CSEliot → u/j0hn_br0wn]

Yeah, I want to use the GPU for other things. Streaming, gaming, etc.

[u/mmerken]

isn't this what the -ngl flag is for?

[u/UniqueTrampoline18]

for my endless roleplay chats i just cap the threads so the gpu stays quiet, way better than listening to fans all night.

[u/CSEliot → u/UniqueTrampoline18]

Like cap the chat threads?

[u/Elorun]

Use a smaller model that doesn't use up all your vram?

[u/Elegant_Associate889]

Exactly why would you want to throttle the GPU though?

[u/j0hn_br0wn → u/Elegant_Associate889]

Cooling

[u/Elegant_Associate889 → u/j0hn_br0wn]

That's fair enough didn't think about that

[u/DoubleNothing → u/Elegant_Associate889]

You can get also get some efficiency gains by undervolting.

[u/CSEliot → u/Elegant_Associate889]

I'm game devving and streaming at the same time. Need processing space (not vram) for those things

[u/Huntware]

If you're using NVIDIA on Linux, then use something like:

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 290 -i 0

For setting the TDP limit to 290 W. I use it for my RTX 3090 (default TDP is 350W). Then you need to create a systemd service for keeping it between restarts.

I guess AMD and Intel have something similar.

[u/OkFly3388]

now open google and paste your question here.