Unraid + GA-Z68AP-D3 IOMMU disabled

[pteriss]

Hei guys!

After being inspired by LTT videos about Unraid, I decided it's my time to dive in there as well. As I had some older hardware easily available (Dell Optiplex 790 sff), I decided to play around on that. I was able to set up IOMMU (1050ti gpu passthrough to a windows vm) on that Dell motherboard with an i5-2500s. But then I decided that I want to upgrade at least parts of my server (case, MB, CPU, additional drives, PSU). So now I have the following setup: 

• i7-2600
• GA-Z68AP-D3 rev 1.0
• 16GB DDR3 RAM
• bunch of drives, ssd's for cache

This is working quite great for me as a home server (seafile, nas, plex, etc), but I also wanted to make a Windows VM with GPU passthrough to be able to do some light gaming on the living room TV. So I made all the cabling (10m hdmi, 10m active usb cables), plugged all in, went into BIOS, made sure the virtualization was enabled, but when booting into unraid, I get this: 

 

 

When googling about this issue with my particular motherboard, I came across this post, which does not seem too optimistic about IOMMU on this boad - link. ;(

I am wondering if anyone has any tips or suggestions on what to actually try before I go about switching my board again. 

 

Best regards,

Peteris

 

[Guest]

Also note that IOMMU in Intel is VT-d, that's what you should enable from the bios, with VT-x

can you get into a linux terminal and type this? dmesg | grep IOMMU

 

Make sure to update the bios too

Edited March 1, 2019 by Guest

[pteriss]

3 minutes ago, Lukyp said:

Also note that IOMMU in Intel is VT-d, that's what you should enable from the bios, with VT-x

can you get into a linux terminal and type this? dmesg | grep IOMMU

 

Make sure to update the bios too

Thanks for the response! dmesg | grep IOMMU returns nothing.  BIOS is the latest available version. I even found that there was some "beta" version available, but QFlash did not allow me to flash that, neither did the msdos flash tool. 

 

EDIT: there is only "virtualization: enabled" in the BIOS. Did not find anything else similar to VT-d or VT-x.

[pteriss]

Full dmesg output here: https://1drv.ms/t/s!Ao8d3jJR_0D1g_g9CrZbFj46k-LAUg

[Guest]

2 minutes ago, pteriss said:

Thanks for the response! dmesg | grep IOMMU returns nothing.  BIOS is the latest available version. I even found that there was some "beta" version available, but QFlash did not allow me to flash that, neither did the msdos flash tool. 

Uhhh really strange, it could be an unsupported chipset for IOMMU in Linux, but since you posted that thread where they talk about crappy BIOS support it, I would just try to update the BIOS from Windows (or either some small Windows environment USB-bootable) or if your bios supports it, directly from it, never owned a gigabyte BIOS tbh

Still you can try from a linux distro like Ubuntu if it detects IOMMU support for it

 

[Guest]

2 minutes ago, pteriss said:

Full dmesg output here: https://1drv.ms/t/s!Ao8d3jJR_0D1g_g9CrZbFj46k-LAUg

The dmesg output is not complete, the most important things start at 0.000000, and got cut off, try to reboot and take a new log

A complete dump from CLI would be dmesg > dmesg-log.txt

Edited March 1, 2019 by Guest

[pteriss]

1 minute ago, Lukyp said:

The dmesg output is not complete, the most important things start at 0.000000, and got cut off, try to reboot and take a new log

A complete dump from CLI would be dmesg > dmesg-log.txt

That is what I did, except I used >> .

The server went down for a reboot now, will post new output ASAP. 

[pteriss]

Here is the new output. 
https://1drv.ms/t/s!Ao8d3jJR_0D1g_g-t_m0xPimwjwXig

 

Not the best messages there though: 

[    8.116667] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    8.116735] AMD IOMMUv2 functionality not available on this system

[Guest]

1 minute ago, pteriss said:

That is what I did, except I used >> .

The server went down for a reboot now, will post new output ASAP. 

Oh yes I noticed that the kernel log buffer exceeds the maximum limited...

 

6 minutes ago, pteriss said:

Here is the new output. 
https://1drv.ms/t/s!Ao8d3jJR_0D1g_g-t_m0xPimwjwXig

 

Not the best messages there though: 

[    8.116667] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    8.116735] AMD IOMMUv2 functionality not available on this system

What the heck? It's trying to use the AMD iommu driver on an intel system for some reason...
Either the motherboard support is really crap or it's a Linux or Unraid bug

I would try to do those things

•  Try an ubuntu live USB and check how the IOMMU driver is loaded here to see if that's an unraid issue
•  Idk how you do that in Unraid, and if it uses grub or not, but you need to edit the kernel parameters and add something like amd_iommu=off and intel_iommu=on they are typically done in traditional linux OS in /etc/default/grub after GRUB_DEFAULT_CMDLINE="quiet splash amd_iommu=off intel_iommu=on", Then you should update the bootloader to automatically append those parameters every time, but I repeat idk how to do that with unraid

[pteriss]

Just now, Lukyp said:

Oh yes I noticed that the kernel log buffer exceeds the maximum limited...

 

What the heck? It's trying to use the AMD iommu driver on an intel system for some reason...
Either the motherboard support is really crap or it's a Linux or Unraid bug

I would try to do those things

•  Try an ubuntu live USB and check how the IOMMU driver is loaded here to see if that's an unraid issue
•   Idk how you do that in Unraid, and if it uses grub or not, but you need to edit the kernel parameters and add something like amd_iommu=off and intel_iommu=on they are typically done in traditional linux OS in /etc/default/grub after GRUB_DEFAULT_CMDLINE="quiet splash amd_iommu=off intel_iommu=on", Then you should update the bootloader to automatically append those parameters every time, but I repeat idk how to do that with unraid

Cool, I'll try these things in the evening. Don't really wanna lock myself out doing this from work. Thanks for the tips! 

[Guest]

1 minute ago, pteriss said:

Cool, I'll try these things in the evening. Don't really wanna lock myself out doing this from work. Thanks for the tips! 

Btw I think unraid is using syslinux, from the archwiki https://wiki.archlinux.org/index.php/kernel_parameters#Syslinux

It's saying

• Press Tab when the menu shows up and add them at the end of the string:

linux /boot/vmlinuz-linux root=/dev/sda3 initrd=/boot/initramfs-linux.img quiet splash 

Press Enter to boot with these parameters.

• To make the change persistent after reboot, edit /boot/syslinux/syslinux.cfg and add them to the APPEND line:

APPEND root=/dev/sda3 quiet splash

So you should be doing that

• Press Tab when the menu shows up and add them at the end of the string:

linux /boot/vmlinuz-linux root=/dev/sda3 initrd=/boot/initramfs-linux.img quiet splash amd_iommu=off intel_iommu=on 

Press Enter to boot with these parameters.

• To make the change persistent after reboot, edit /boot/syslinux/syslinux.cfg and add them to the APPEND line:

APPEND root=/dev/sda3 quiet splash amd_iommu=off intel_iommu=on 

 

Just be careful of the syntax and do not remove any pre-existing parameters

[pteriss]

Oh man, the unspeakable things I had to do to get this to work! But I'm glad that I did. 

Basically - these parameters did not really do the trick, I got desperate, went through some old sketchy overclocking forums and found some custom modified UEFI image (my board had only BIOS), that was taken from rev 2 of the board and made to work with the rev 1 I have and flashed that. And voila - the UEFI actually had a VT-d setting. Now everything works

 

 

Thanks for the help anyway!