Can you pass through hardrives with proxmox

[PurpleCodes]

I've seen this done with HBA's and Raid cards but can you do this with the sata ports on the motherboard?

 

I am using a ASUS AM4 TUF GAMING X570-Plus with a ryzen 2600

 

I want to pass in the hardrives to use with freenas and I want to make sure it's possible.

 

https://www.newegg.com/asus-tuf-gaming-x570-plus/p/N82E16813119198?Item=N82E16813119198&nm_mc=AFC-RAN-COM&cm_mmc=AFC-RAN-COM&utm_medium=affiliates&utm_source=afc-PCPartPicker&AFFID=2558510&AFFNAME=PCPartPicker&ACRID=1&ASID=https%3a%2f%2fpcpartpicker.com%2fproduct%2fwhMTwP%2fasus-tuf-gaming-x570-plus-atx-am4-motherboard-tuf-gaming-x570-plus&ranMID=44583&ranEAID=2558510&ranSiteID=8BacdVP0GFs-rEk1CVIwod3ELFpHG4_zSg

[BrinkGG]

15 minutes ago, PurpleCodes said:

I've seen this done with HBA's and Raid cards but can you do this with the sata ports on the motherboard?

 

I am using a ASUS AM4 TUF GAMING X570-Plus with a ryzen 2600

 

I want to pass in the hardrives to use with freenas and I want to make sure it's possible.

 

https://www.newegg.com/asus-tuf-gaming-x570-plus/p/N82E16813119198?Item=N82E16813119198&nm_mc=AFC-RAN-COM&cm_mmc=AFC-RAN-COM&utm_medium=affiliates&utm_source=afc-PCPartPicker&AFFID=2558510&AFFNAME=PCPartPicker&ACRID=1&ASID=https%3a%2f%2fpcpartpicker.com%2fproduct%2fwhMTwP%2fasus-tuf-gaming-x570-plus-atx-am4-motherboard-tuf-gaming-x570-plus&ranMID=44583&ranEAID=2558510&ranSiteID=8BacdVP0GFs-rEk1CVIwod3ELFpHG4_zSg

Yes! This is exactly the setup I use on my homelab. 

This quick reference tells you the basics of doing it. https://pve.proxmox.com/wiki/Physical_disk_to_kvm

You can also enter the Qemu config file for the VM (once created) and add them as sata drives by Serial number. 

[PurpleCodes]

4 minutes ago, BrinkGG said:

Yes! This is exactly the setup I use on my homelab. 

This quick reference tells you the basics of doing it. https://pve.proxmox.com/wiki/Physical_disk_to_kvm

You can also enter the Qemu config file for the VM (once created) and add them as sata drives by Serial number. 

Thanks so much, You have no idea how good it feels to be right for once. Been specing out my server for days now and everyone is telling my different ways to do stuff. Just glad i'm on the right track.

[BrinkGG]

2 minutes ago, PurpleCodes said:

Thanks so much, You have no idea how good it feels to be right for once. Been specing out my server for days now and everyone is telling my different ways to do stuff. Just glad i'm on the right track.

As long as you don't put the drives in hardware raid (raid card), you'll have no issue with the hardware you mentioned.  

[PurpleCodes]

34 minutes ago, BrinkGG said:

As long as you don't put the drives in hardware raid (raid card), you'll have no issue with the hardware you mentioned.  

If i wanted a raid card how would that work? just curious

[Electronics Wizardy]

2 hours ago, PurpleCodes said:

If i wanted a raid card how would that work? just curious

Depends on your goals. Normally the host will have a filesystem and give virtual disks to the vms

 

Also why do you want to passthough disks? Normally its better to setup raid and stoarge on the proxmox host, then passthough the virtual disks to the vms.

[PurpleCodes]

22 hours ago, Electronics Wizardy said:

Depends on your goals. Normally the host will have a filesystem and give virtual disks to the vms

 

Also why do you want to passthough disks? Normally its better to setup raid and stoarge on the proxmox host, then passthough the virtual disks to the vms.

From what i've read and heard sending virtual drives to freenas kills preformance. In an ideal world I would have 2 servers and Just have freenas run on one of them and a hypervisor on another but I don't really have a budget for that.

[Electronics Wizardy]

29 minutes ago, PurpleCodes said:

From what i've read and heard sending virtual drives to freenas kills preformance. In an ideal world I would have 2 servers and Just have freenas run on one of them and a hypervisor on another but I don't really have a budget for that.

why run freenas at all? You can setup a zfs share on the host, so the main point of freenas is removed. THen just run a conatianer for stuff like smb shares.

[PurpleCodes]

18 minutes ago, Electronics Wizardy said:

why run freenas at all? You can setup a zfs share on the host, so the main point of freenas is removed. THen just run a conatianer for stuff like smb shares.

What would the difficulty of adding things like plex and encrypted backup? Freenas makes it really easy to set that stuff up how would I go about doing that with proxmox and containers?

[TehDwonz]

On 7/14/2020 at 9:09 PM, PurpleCodes said:

I've seen this done with HBA's and Raid cards but can you do this with the sata ports on the motherboard?

 

I am using a ASUS AM4 TUF GAMING X570-Plus with a ryzen 2600

 

I want to pass in the hardrives to use with freenas and I want to make sure it's possible.

You can, as already mentioned - but it might be better to pass through the whole controller if you must use FreeNAS, unless you have non NAS disks on the same controller?
It's odd, I just had to do exactly what you wanted today, for a single disk - I'm waiting on a new SFF8087 to SATA cable, so temporarily passed a drive through from the motherboard controller, while the rest were on half of a SAS card.

But for the most part, it is indeed better to set up the physical disks on the hypervisor, and allocate virtual ones to your VMs. Just be aware that ZFS on top of RAID does not play nice! If you go ZFS, put your controllers in IT mode and let ZFS handle it all in software.

[PurpleCodes]

2 minutes ago, TehDwonz said:

You can, as already mentioned - but it might be better to pass through the whole controller if you must use FreeNAS, unless you have non NAS disks on the same controller?
It's odd, I just had to do exactly what you wanted today, for a single disk - I'm waiting on a new SFF8087 to SATA cable, so temporarily passed a drive through from the motherboard controller, while the rest were on half of a SAS card.

But for the most part, it is indeed better to set up the physical disks on the hypervisor, and allocate virtual ones to your VMs. Just be aware that ZFS on top of RAID does not play nice! If you go ZFS, put your controllers in IT mode and let ZFS handle it all in software.

So i just bought a HBA card (https://www.amazon.com/SAS9211-8I-8PORT-Int-Sata-Pcie/dp/B002RL8I7M/ref=as_li_ss_tl?keywords=SAS9211&qid=1577977693&sr=8-1&linkCode=sl1&tag=craftcomputin-20&linkId=ef0870a4aafb2f7b0e72e5765f068e62&language=en_US)

But I am not 100% sure that my mobo supports the pcie pass through. I know the ryzen cpu does but I guess it's a gamble on if it is going to work. I have 2 other drives (m.2) which I am going to run in a raid 1 for all of my vms.

[TehDwonz]

2 minutes ago, PurpleCodes said:

So i just bought a HBA card (https://www.amazon.com/SAS9211-8I-8PORT-Int-Sata-Pcie/dp/B002RL8I7M/ref=as_li_ss_tl?keywords=SAS9211&qid=1577977693&sr=8-1&linkCode=sl1&tag=craftcomputin-20&linkId=ef0870a4aafb2f7b0e72e5765f068e62&language=en_US)

But I am not 100% sure that my mobo supports the pcie pass through. I know the ryzen cpu does but I guess it's a gamble on if it is going to work. I have 2 other drives (m.2) which I am going to run in a raid 1 for all of my vms.

The BIOS feature you are looking for is "IOMMU".

[PurpleCodes]

Just now, TehDwonz said:

The BIOS feature you are looking for is "IOMMU".

Ya I looked for that, Just now found it actually and it does support it, so it looks like i can pass my entire hba into freenas (assuming I still do freenas)

[TehDwonz]

Just now, PurpleCodes said:

Ya I looked for that, Just now found it actually and it does support it, so it looks like i can pass my entire hba into freenas (assuming I still do freenas)

Be aware that IOMMU "groupings" can mean passthrough is somewhat limited. In that you can't pass 2 devices in the same group to 2 different VMs.
There is a reasonable discussion on this (spit) reddit: https://www.reddit.com/r/VFIO/comments/dq5w2o/what_are_some_suitable_x570_motherboards/

 

[PurpleCodes]

Just now, TehDwonz said:

Be aware that IOMMU "groupings" can mean passthrough is somewhat limited. In that you can't pass 2 devices in the same group to 2 different VMs.
There is a reasonable discussion on this (spit) reddit: https://www.reddit.com/r/VFIO/comments/dq5w2o/what_are_some_suitable_x570_motherboards/

 

Well i only plan to passthrough the hba card into freenas and nothing else.

[TehDwonz]

Just now, PurpleCodes said:

Well i only plan to passthrough the hba card into freenas and nothing else.

Ok, but don't forget you have on-board PCIe/nvme devices that might share groupings with certain slots. The top x16 is usually on its own, but not always. It's an annoyingly undocumented thing that griefs AMD chipsets in particular.

[PurpleCodes]

1 minute ago, TehDwonz said:

Ok, but don't forget you have on-board PCIe/nvme devices that might share groupings with certain slots. The top x16 is usually on its own, but not always. It's an annoyingly undocumented thing that griefs AMD chipsets in particular.

Well i'll hope for the best. I saw a video on craft computing that allowed him to pass through the individual drives from the hba card instead of the actual hba card. Is this still using IOMMU or is it something else? How do you know if a board does IOMMU or IOMMU groupings. this is my board https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM4/PRIME_X570-P/E15829_PRIME_PRO_TUF_GAMING_X570_Series_BIOS_EM_WEB.pdf

[Electronics Wizardy]

53 minutes ago, PurpleCodes said:

What would the difficulty of adding things like plex and encrypted backup? Freenas makes it really easy to set that stuff up how would I go about doing that with proxmox and containers?

Not really. You can do zfs encryption on the host, and you can setup a container for plex as well(its nice to keep it separated).

 

I don't see a reason to run freenas here at all, and this would make it much simpler and easier to work with

[PurpleCodes]

1 minute ago, Electronics Wizardy said:

Not really. You can do zfs encryption on the host, and you can setup a container for plex as well(its nice to keep it separated).

 

I don't see a reason to run freenas here at all, and this would make it much simpler and easier to work with

As said earlier in the thread because of IOMMU groupings it might not be possible to pass the drives into multiple vms. unless a container doesn't count not entirely sure

[Electronics Wizardy]

1 minute ago, PurpleCodes said:

As said earlier in the thread because of IOMMU groupings it might not be possible to pass the drives into multiple vms. unless a container doesn't count not entirely sure

your not passing the disks to the container, your passing a mount point, and that will work on any system.

 

I just do the file sharing on my proxmox system on the host, but thats does take some of the advanages ouf vms out, so thats why I suggest using a conatiner for doing file sharing

[TehDwonz]

9 minutes ago, PurpleCodes said:

As said earlier in the thread because of IOMMU groupings it might not be possible to pass the drives into multiple vms. unless a container doesn't count not entirely sure

You can pass individual disks no problem - the grouping issue only comes up when passing through controllers/PCI devices. I don't think it'll be an issue for you tbh, as @Electronics Wizardy said. Let ProxMox handle the physical hardware, and pass virtual drives / mount points to VMs and containers. It makes the VMs hardware agnostic, so they can more easily be moved should your host system fail. If you tie them to actual hardware, you'll have to fix it all manually.

 

[tekdmon]

Good info in this thread.  I am getting ready to setup a Proxmox server and I see (3) methods to use HDDS.

 

1.) Setup ZFS on Proxmox and pass a mount point of the drive(s) to the VM

2.) Pass through individual drive(s) to VM

3.) Pass through SATA controller or HBA card to VM

 

What are the pros and cons to each?  It seems like #1 is the easiest, since the disk are setup in Proxmox.   I plan on setting up a NAS VM or container to manage files (Turnkey Linux File Server, TrueNAS, or Open Media Vault.)