(ACS Override) I need to patch my kernal, but I've never done it before

starry · December 15, 2020

I need to break up these IOMMU groups so that the Nvidia stuff is isolated from everything else.

Here is a list of all my groups and what they contain:

Spoiler



IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06)
IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
        00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] [1002:67ef] (rev cf)
        01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
        02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] [10de:21c4] (rev a1)
        02:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
        02:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
        02:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)
IOMMU Group 10:
        00:1f.0 ISA bridge [0601]: Intel Corporation Z97 Chipset LPC Controller [8086:8cc4]
        00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
        00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
IOMMU Group 11:
        06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
IOMMU Group 2:
        00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
IOMMU Group 3:
        00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
IOMMU Group 4:
        00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]
IOMMU Group 5:
        00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
IOMMU Group 6:
        00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)
IOMMU Group 7:
        00:1c.2 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev d0)
        04:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 03)
IOMMU Group 8:
        00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)
IOMMU Group 9:
        00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]

There are only 2 PCIE 16x slots on my motherboard and they are both in the same group, there is also a 1x slot that I know has its own group, but its got a physical cap on it so i cant plug in anything bigger, so it looks like a kernel patch is my only option here.

I am running Manjaro Linux, and so far most of the guides Ive found on how to do this have been for Debian distributions. I tried to follow the Arch Wiki but I dont know enough about the kernal or building things manually to figure it out myself, so posting here is all I could think of.

Here is what I was trying to follow along with on the Arch Wiki:

Spoiler

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Bypassing_the_IOMMU_groups_(ACS_override_patch)

I have only a very vague idea of what its asking me to do. Im a pretty slow learner so Im still trying to figure out this linux stuff.

Thank you very much, I really appreciate any help. If anyone needs anymore information, please let me know and Ill be happy to provide it.

starry · December 15, 2020

I guess I should probably pose a question if someone is going to answer me.

As a first step, how should I go about patching the kernel?

Nayr438 · December 15, 2020

You could install the suggested kernel that already has the patch applied. https://aur.archlinux.org/packages/linux-vfio/

Otherwise you need to merge this patch with the kernel source https://aur.archlinux.org/cgit/aur.git/plain/add-acs-overrides.patch?h=linux-vfio

From the the source you can use the patch command to merge the changes

You can refer to https://www.pair.com/support/kb/paircloud-diff-and-patch/

Then add "pcie_acs_override=downstream,multifunction" to your kernel command line.

starry · December 15, 2020

Just now, Nayr438 said:

You could install the suggested kernel that already has the patch applied. https://aur.archlinux.org/packages/linux-vfio/

Otherwise you need to merge this patch with the kernel source https://aur.archlinux.org/cgit/aur.git/plain/add-acs-overrides.patch?h=linux-vfio

From the the source you can use the patch command to merge the changes

You can refer to https://www.pair.com/support/kb/paircloud-diff-and-patch/

Then add "pcie_acs_override=downstream,multifunction" to your kernel command line.

Thank you very much, but Im not really sure how install a new kernel, could you elaborate on how to do that? And can you also explain what a the term "kernal command line" means? Thanks again, Im sorry if those seem like silly questions.

I did see the VFIO package on AUR but I wasnt really sure how to install it. Ive never touched the actual linux kernal before so Im kind of lost here, and maybe a bit nervous about breaking my installation. Googling how to install a new kernal from a package only brings up results on how to update to the latest official linux kernal, but that just might be my poor web searching skills. I did find one page that explained what I wanted, but it was published in 2010, and if Ive learned anything using linux for the past year, its to not follow super old guides.

Thank you so much for your reply and help.

starry · December 15, 2020

Update: Ive gone through a lot of reading, I figured out that all I need to do is use makepkg like a normal package. Its building right now, for some reason I was expecting it to be something else. I hope it goes well. If I break my install feel free to facepalm.

starry · December 15, 2020

Starting to get worried, its suddenly spitting out lots of errors (cropped to hide username)

starry · December 15, 2020

It aborted itself and now when I try to run it again it returns this:

makepkg -si                     
==> Making package: linux-vfio 5.9.14.arch1-1 (Tue 15 Dec 2020 09:50:25 AM EST)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Updating archlinux-linux git repo...
Fetching origin
  -> Found config
  -> Found add-acs-overrides.patch
  -> Found i915-vga-arbiter.patch
  -> Found sphinx-workaround.patch
==> Validating source files with sha256sums...
    archlinux-linux ... Skipped
    config ... Passed
    add-acs-overrides.patch ... Passed
    i915-vga-arbiter.patch ... Passed
    sphinx-workaround.patch ... Passed
==> Verifying source file signatures with gpg...
    archlinux-linux git repo ... Passed
==> Extracting sources...
  -> Creating working copy of archlinux-linux git repo...
Reset branch 'makepkg'
==> Starting prepare()...
Setting version...
Applying patch add-acs-overrides.patch...
patching file Documentation/admin-guide/kernel-parameters.txt
Hunk #1 succeeded at 3600 (offset 32 lines).
patching file drivers/pci/quirks.c
Hunk #2 succeeded at 4926 (offset 2 lines).
Applying patch i915-vga-arbiter.patch...
patching file drivers/gpu/drm/i915/display/intel_display.c
Hunk #1 succeeded at 18795 (offset 130 lines).
patching file drivers/gpu/drm/i915/display/intel_display.h
Hunk #1 succeeded at 514 (offset 7 lines).
patching file drivers/gpu/drm/i915/display/intel_vga.c
patching file drivers/gpu/drm/i915/display/intel_vga.h
patching file drivers/gpu/drm/i915/i915_drv.c
patching file drivers/gpu/drm/i915/i915_params.c
Hunk #1 succeeded at 135 with fuzz 2.
patching file drivers/gpu/drm/i915/i915_params.h
Hunk #1 succeeded at 71 (offset 1 line).
Applying patch sphinx-workaround.patch...
patching file Documentation/conf.py
Hunk #1 succeeded at 36 with fuzz 2.
Setting config...
scripts/kconfig/conf  --olddefconfig Kconfig
#
# No change to .config
#

  ERROR: Kernel configuration is invalid.
         include/generated/autoconf.h or include/config/auto.conf are missing.
         Run 'make oldconfig && make prepare' on kernel src to fix it.

make: *** [Makefile:719: include/config/auto.conf] Error 1
==> ERROR: A failure occurred in prepare().
    Aborting...

Running make oldconfig && make prepare like it says does nothing

starry · December 15, 2020

I got it!

Sorry for making so many replies to this post, I wanted to keep it updated because I was having such a hard time figuring it out and any advice was amazing.

IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06)
IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 10:
        00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]
IOMMU Group 11:
        00:1f.0 ISA bridge [0601]: Intel Corporation Z97 Chipset LPC Controller [8086:8cc4]
        00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
        00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
IOMMU Group 12:
        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] [1002:67ef] (rev cf)
IOMMU Group 13:
        01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
IOMMU Group 14:
        02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] [10de:21c4] (rev a1)
IOMMU Group 15:
        02:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
IOMMU Group 16:
        02:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
IOMMU Group 17:
        02:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)
IOMMU Group 18:
        04:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 03)
IOMMU Group 19:
        06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
IOMMU Group 2:
        00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
IOMMU Group 3:
        00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
IOMMU Group 4:
        00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
IOMMU Group 5:
        00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]
IOMMU Group 6:
        00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
IOMMU Group 7:
        00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)
IOMMU Group 8:
        00:1c.2 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev d0)
IOMMU Group 9:
        00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)

This was a very long learning experience. Thank you @Nayr438for helping me get on the right track.

Nayr438 · December 15, 2020

Sorry was at work. Glad to hear you got it going.

starry · December 15, 2020

8 minutes ago, Nayr438 said:

Sorry was at work. Glad to hear you got it going.

No worries man, I appreciate the help.

I seem to have another problem. One of the steps on the wiki to set my devices up for pass through has me run this command, which returns an error:

sudo mkinitcpio -p linux59
==> Building image from preset: /etc/mkinitcpio.d/linux59.preset: 'default'
  -> -k /boot/vmlinuz-5.9-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.9-x86_64.img
==> ERROR: '/lib/modules/5.9.11-3-MANJARO' is not a valid kernel module directory
==> Building image from preset: /etc/mkinitcpio.d/linux59.preset: 'fallback'
  -> -k /boot/vmlinuz-5.9-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.9-x86_64-fallback.img -S autodetect
==> ERROR: '/lib/modules/5.9.11-3-MANJARO' is not a valid kernel module directory

I think something somewhere hasnt update after manually patching the kernal. Running uname returns this:

Linux Tempest 5.9.14-arch1-1-vfio #1 SMP PREEMPT Mon, 14 Dec 2020 04:15:14 +0000 x86_64 GNU/Linux

But I noticed that in my settings manager it says Im running kernel version 5.9.11-3, not 5.9.14:

This might not be related to my issue, but my best guess says it is. Do you have any idea how I could go about fixing this? Thank you very much for your time.

Nayr438 · December 15, 2020

Whats the output of "ls /etc/mkinitcpio.d/" and "pacman -Q | grep linux"

starry · December 15, 2020

1 minute ago, Nayr438 said:

Whats the output of "ls /etc/mkinitcpio.d/" and "pacman -Q | grep linux"

$ls /etc/mkinitcpio.d/
linux59.preset  linux-vfio.preset

$pacman -Q | grep linux
archlinux-appstream-data 20201128-1
archlinux-keyring 20201028-1
lib32-util-linux 2.36.1-1
linux-api-headers 5.8-1
linux-firmware 20201124.r1786.b362fd4-1
linux-vfio 5.9.14.arch1-1
linux59 5.9.11-3
util-linux 2.36.1-3
util-linux-libs 2.36.1-3

Nayr438 · December 15, 2020

I would ignore Manjaro's kernel manager then. I am honestly not sure how it works as I run Arch and that's Manjaro Specific.

If uname says "5.9.14-arch1-1-vfio" then your on the vfio kernel and that's all that matters.

Are your trying to rebuild initramfs for the Manjaro kernel or vfio?

For vfio it should be "sudo mkinitcpio -p linux-vfio" or you can regenerate it for all the kernel presets with "sudo mkinitcpio -P"

For linux59 try reinstalling it , "pacman -S linux59" and check the output of "ls /lib/modules/" afterwards.

starry · December 15, 2020

24 minutes ago, Nayr438 said:

This is the step that I was on:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Isolating_the_GPU

After finishing the two things above it, it instructs me to "regenerate the initramfs", which is formatted as a hyperlink that takes me to this page:

https://wiki.archlinux.org/index.php/Mkinitcpio#Image_creation_and_activation

Because the instructions on manual generation arent specific to setting up PCI passthrough I was hesitant to use "-P", as I am also following along with a video guide where he goes through and explains each section of this wiki page in order with a demonstration, and he used "mkinitcpio -p linux54" (where 54 is his kernel version). The difference was that he didnt have to patch his kernel because his IOMMU groups were already set up very nicely. So should I assume that using "mkinitcpio -P" will do just fine for that purpose?

Nayr438 · December 15, 2020

"mkinitcpio -P" should work fine assuming the Manjaro kernel and Arch Kernel are built the same.

The presets located in "/etc/mkinitcpio.d/" are what mkinitcpio will build from. You will have a preset for each kernel.

Selecting "-P" will build for all presets in that directory or specifying "-p preset" will just build for that preset.

If your modifying mkinitcpio.conf rather than a separate configuration file, then eventually all your kernel initramfs images will end up being rebuilt with it at some point in the event they update.

starry · December 15, 2020

Just now, Nayr438 said:

"mkinitcpio -P" should work fine assuming the Manjaro kernel and Arch Kernel are built the same.

The presets located in "/etc/mkinitcpio.d/" are what mkinitcpio will build from. You will have a preset for each kernel.

Selecting "-P" will build for all presets in that directory or specifying "-p preset" will just build for that preset.

If your modifying mkinitcpio.conf rather than a separate configuration file, then eventually all your kernel initramfs images will end up being rebuilt with it at some point in the event they update.

Thank you so much! As far as I'm aware manjaro is just the arch kernal but it has a bunch of stuff already built up so that its more like a prepackaged distro for the average user, so I think it should work just fine.

starry · December 15, 2020

10 minutes ago, Nayr438 said:

"mkinitcpio -P" should work fine assuming the Manjaro kernel and Arch Kernel are built the same.

The presets located in "/etc/mkinitcpio.d/" are what mkinitcpio will build from. You will have a preset for each kernel.

Selecting "-P" will build for all presets in that directory or specifying "-p preset" will just build for that preset.

If your modifying mkinitcpio.conf rather than a separate configuration file, then eventually all your kernel initramfs images will end up being rebuilt with it at some point in the event they update.

Thanks it worked!

starry · December 16, 2020

I knew exactly what I was trying to set up but it still blows my mind that I ran games on a virtual machine last night. Fantastic.

gardotd426 · December 25, 2020

git clone https://github.com/frogging-family/linux-tkg && cd linux-tkg

nano customization.cfg

On the acs override line, make sure you put "true".

Ctrl+S to save, Ctrl+X to exit.

makepkg -si

sudo update-grub

Done. You now have an acs-patched kernel.

starry · December 25, 2020

13 hours ago, gardotd426 said:
git clone https://github.com/frogging-family/linux-tkg && cd linux-tkg
nano customization.cfg
On the acs override line, make sure you put "true".

Ctrl+S to save, Ctrl+X to exit.
makepkg -si
sudo update-grub
Done. You now have an acs-patched kernel.

Thanks but i already got it working. Appreciate the suggestion though ^^

Sign In

(ACS Override) I need to patch my kernal, but I've never done it before

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites