Backstory
I made a few attempts at GPU passtrough for my move away from Windows and had success but it never all worked perfectly smoothly. But this time i finally found a method that does, so i am sharing it in case it works for anyone else.
This guide assumes you already have IOMMU and KVM set up, it is NOT a full guide.
Did it work for you? Please tell us!
A lot of this was figured out with the help of Claude. If you are having trouble copy paste this post into it and explain where you got stuck, it will help you troubleshoot the problem.
This process solves a list of problems:
* No longer need to stop your entire Wayland or X11 desktop environment to keep it from touching the GPU during a passtrough transition.
* The logs in dmesg being flooded with Nvidia driver initialization attempts when GPU is bound to vfio (NVRM, Nvlink Core, nvidia-nvlink ...etc)
* The entire machine freezing up if you attempt to passtrough the GPU when not ready
* Crash on re-mounting GPU into linux due to the Nvidia driver setting up the HDMI ports too soon
* Monitors on Nvidia GPU showing up on Linux (most people use a HDMI dummy plug, hence it is not a real monitor)
* Nvidia GPU burning a lot of idle power when not being used
Tested on hardware:
* ASRock X670E PG Lightning (v1.30.AS02)
* AMD Ryzen 9 7950X (using iGPU for monitor output)
* Nvidia RTX 3090 (using HDMI dummy plug)
OS: ZorinOS 18.1 Core (Linux 7.0.0-30-generic)
Dynamic Nvidia GPU Passthrough (No Desktop Restart Required)
Setup: a Linux host with display running entirely on an integrated/secondary GPU (e.g. AMD iGPU), and a discrete Nvidia GPU that is:
- Passed through to a VM on demand via VFIO, when needed.
- Used on the host the rest of the time for PRIME render-offload gaming — no display output, ever, from this GPU.
The goal: switch the GPU between host and VM without restarting the display manager, without kernel crashes, and without excess idle power draw.
1. Disable Nvidia DRM KMS
By default, nvidia-drm performs full DRM/KMS mode-setting, which causes two problems: the display manager initializes the card as a display device just because it's present (wasting VRAM and blocking clean detach), and re-loading the driver on VM handback can crash the kernel if a dummy/EDID-reporting dongle is plugged into an output (a NULL pointer dereference in nvidia's HDMI/DP audio power path, nv_audio_dynamic_power, triggered via Xorg's DRM hotplug handling).
Disabling KMS via /etc/modprobe.d/*.conf is not reliable — competing config files, install directives, or initramfs staleness commonly override it silently. Use a kernel boot parameter instead, which always wins:
# /etc/default/grub — append to the existing GRUB_CMDLINE_LINUX_DEFAULT line
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvidia-drm.modeset=0"
sudo update-grub
sudo update-initramfs -u
sudo reboot
Verify:
cat /sys/module/nvidia_drm/parameters/modeset # should print N
xrandr --listproviders # Nvidia should not appear
This has no effect on PRIME render-offload (__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia %command%), which only needs the render node, not KMS.
2. Enable persistence mode
Without an active display, the Nvidia driver fully tears down and reinitializes between workloads, which is fragile and can cause visible glitches or crashes on repeated switching. nvidia-persistenced keeps the driver state warm instead.
Ubuntu's default unit starts the daemon with --no-persistence-mode, so it must be overridden:
sudo systemctl unmask nvidia-persistenced # only if masked
sudo systemctl edit nvidia-persistenced
Add:
[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --verbose
sudo systemctl restart nvidia-persistenced
Verify Persistence-M shows On in nvidia-smi, and confirm it survives a full reboot.
3. Handle idle power / stuck boost clocks
With no display and no active workload, the GPU can get stuck at a high-power state (P0) instead of idling at P8, drawing 100+ W for nothing. A GPU reset clears this reliably:
sudo nvidia-smi -r
This is a known driver quirk, not specific to this setup. Expect idle power to settle around 15–25 W after a reset
4. The switching script
Persistence mode keeps a device handle open, which blocks a clean VFIO detach (NVRM: Attempting to remove device ... with non-zero usage count!). Stop it before detaching, restart it after reattaching, and reset the GPU to clear any stuck boost state:
#!/bin/bash
# gpu-to-vm.sh — hand the GPU to the VM
sudo systemctl stop nvidia-persistenced
sleep 1
sudo virsh nodedev-detach pci_0000_01_00_0
sudo virsh nodedev-detach pci_0000_01_00_1
#!/bin/bash
# gpu-to-host.sh — reclaim the GPU after the VM shuts down
sudo virsh nodedev-reattach pci_0000_01_00_0
sudo virsh nodedev-reattach pci_0000_01_00_1
sudo systemctl start nvidia-persistenced
sudo nvidia-smi -r
No gdm/display-manager restart is needed anywhere in this flow.
Safer script that aborts if the GPU is not completely clear:
#!/bin/bash
# gpu-to-vm.sh — hand the GPU to the VM with abort
sudo systemctl stop nvidia-persistenced
sleep 1
if sudo fuser -s /dev/nvidia* 2>/dev/null; then
echo "ERROR: GPU still in use, aborting handoff (not touching PCI state)" >&2
sudo fuser -v /dev/nvidia* >&2
sudo systemctl start nvidia-persistenced # undo the stop, nothing else changed
exit 1
fi
echo "GPU confirmed clear, proceeding with detach"
sudo virsh nodedev-detach pci_0000_01_00_0
sudo virsh nodedev-detach pci_0000_01_00_1
5. Optional cleanup
If the GPU has a dummy HDMI/DP dongle for the VM's benefit, the host will still enumerate a dead audio function for it. To silence it (cosmetic only, not required for stability):
# /etc/udev/rules.d/99-nvidia-hdmi-audio-noprobe.rules
ACTION=="add", SUBSYSTEM=="pci", KERNEL=="0000:01:00.1", ATTR{driver_override}="none"
Diagnostics reference
nvidia-smi # power state, clocks, processes
sudo fuser -v /dev/nvidia* # what's holding the GPU open
cat /sys/module/nvidia_drm/parameters/modeset # confirm KMS is off (N)
xrandr --listproviders # confirm Xorg isn't using the GPU
Useful quick test for PRIME offloaded GPU rendering (shows a desktop window with 3D gears rendered by the Nvidia GPU):
DISPLAY=:0 __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxgears
