In this tutorial we're going to take a look at how to pass a GPU to a Windows QEMU VM on an Arch linux host, and to do so we're going to need a host PC with 2 graphics cards. the first graphics card is for the host OS, and the second graphics card is for the VM itself.
First change the mkinitcpio.conf file, as instructed here in order to enable vfio and also grub, in order to enable iommu for intel cpus:
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ sudo vim /etc/default/grub
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ cat /etc/default/grub | grep intel
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 intel_iommu=on iommu=pt"
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ sudo grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/intel-ucode.img /boot/initramfs-linux.img
Found fallback initrd image(s) in /boot: intel-ucode.img initramfs-linux-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Found Windows Boot Manager on /dev/nvme0n1p1@/EFI/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for UEFI Firmware Settings ...
done
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ sudo vim /etc/mkinitcpio.conf
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ cat /etc/mkinitcpio.conf | grep vfio
MODULES=( nvme vmw_pvscsi vfio_pci vfio vfio_iommu_type1 )
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ sudo mkinitcpio -P
[sudo] password for nothing:
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
then rebotot to check if you have vfio:
[ 10.66.66.2/32 ] [ /dev/pts/1 ] [~]
→ sudo reboot now
[ 10.66.66.2/32 ] [ /dev/pts/1 ] [~]
→ sudo dmesg | grep IOMMU
[sudo] password for nothing:
[ 0.018630] DMAR: IOMMU enabled
[ 1.158900] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel jroedel@suse.de
[ 1.158901] AMD-Vi: AMD IOMMUv2 functionality not available on this system
[ 10.66.66.2/32 ] [ /dev/pts/1 ] [~]
→ lspci -nn | grep -iP "VGA|audio"
00:1f.3 Audio device [0403]: Intel Corporation Tiger Lake-H HD Audio Controller [8086:43c8] (rev 11)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] [10de:2482] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1)
04:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
Here you can see the 2 GPUs i currently have, one is a GTX 1050, the other is a RTX 3070. We're going to pass the GT 1050 to our VM. So to do that we need the Device ID this is the following:
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1)
04:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
Now since we're on an Arch based distro we're going to edit the kernel parameters to group the gtx 1050 ids within the vfio parameter. With this we will ISOLATE the GPU, meaning it will not be usable on the host until we reverse the manipulation, of course i have my second GPU, the 3070 to make sure i can still use my setup.
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ cat /etc/default/grub | grep vfio
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=10de:1c81,10de:0fb9"
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ sudo grub-mkconfig -o /boot/grub/grub.cfg
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ cat /boot/grub/grub.cfg | grep vfio
linux /vmlinuz-linux root=UUID=64472324-3a33-44f1-87f7-95431d6946ef rw loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=10de:1c81,10de:0fb9
linux /vmlinuz-linux root=UUID=64472324-3a33-44f1-87f7-95431d6946ef rw loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=10de:1c81,10de:0fb9
linux /vmlinuz-linux root=UUID=64472324-3a33-44f1-87f7-95431d6946ef rw loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=10de:1c81,10de:0fb9
Now as you can see, after editing grub again, to add the ids of the gpu we want to isolate, we generated the grub config again, and in grub.cfg, vfio has been specified as intended, so we reboot and we should NOT have access to the isolated gpu anymore, be very careful to have a second gpu ready so you can still use your system !!!
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ neofetch
-------------------------------------- nothing@nowhere
-------------------------------------- ---------------
-------------------------------------- OS: Bedrock Linux 0.7.27 Poki x86_64
---\\\\\\\\\\\\----------------------- Host: MS-7D09 1.0
----\\\ \\\---------------------- Kernel: 5.18.14-arch1-1
-----\\\ \\\--------------------- Uptime: 6 mins
------\\\ \\\\\\\\\\\\\\\\\------ Packages: 1383 (pacman), 3065 (dpkg), 800 (rpm), 58 (xbps-query), 455 (emerge), 115 (nix-user)
-------\\\ \\\----- Shell: zsh 5.9
--------\\\ \\\---- Resolution: 1920x1080, 3840x2160, 1920x1080, 3840x2160
---------\\\ ______ \\\--- WM: i3
----------\\\ ///--- Theme: Numix-BLACK-SLATE [GTK2/3]
-----------\\\ ///---- Icons: Flat-Remix-Blue-Dark [GTK2/3]
------------\\\ ///----- Terminal: tmux
-------------\\\////////////////------ CPU: 11th Gen Intel i7-11700K (16) @ 5.100GHz
-------------------------------------- GPU: NVIDIA GeForce RTX 3070 Ti <------- that is the one i will use !!!
-------------------------------------- GPU: NVIDIA GeForce GTX 1050<------------------ that is the one we will isolate !!!
-------------------------------------- Memory: 3203MiB / 31998MiB
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ sudo reboot now
Currently, when you reboot your display will be broken past the "starting version arch" line, do not be alarmed. Idk how to fix that, but you can blindly type in your luks password and it will boot just fine.
now after rebooting, we can check that we can use the GPU like so:
[ 10.8.0.4/24 ] [ nowhere ] [Github/blog/servers]
→ sudo virt-manager
[sudo] password for nothing:
so from here we are able to add a GPU to a VM.
add both PCI devices (one is video the other is sound)
then on the VM you need to install the geforce experience drivers:
make sure you do Custom install and tick everything, so it will REMOVE The existing GPU driver to install it again, and from there you should be good to go you can check that the gpu is correctly detected from teh device manager:
And that's it! you can now monitor your gpu usage from inside the VM:
Donate XMR: 8AUYjhQeG3D5aodJDtqG499N5jXXM71gYKD8LgSsFB9BUV1o7muLv3DXHoydRTK4SZaaUBq4EAUqpZHLrX2VZLH71Jrd9k8
Contact: nihilist@contact.nowhere.moe (PGP)
