FreeNAS or Unraid via virtual machine

[Strikermed]

Hi all,

 

I'm building a new server to utilize windows server 2012 r2 for many reasons.  Mostly bc I want the flexibility of the windows environment that I'm comfortable using.  That being said, I also want to learn more about server management and the capabilities of a domain controller.

 

the reason for this thread is to Get some experienced feedback.  I want to use a VM to run a freenas system in a home production environment.  At first I wanted to test the function of freenas, but now I'm having to rely on it as a backup due to my LSI 9266 8i card acting up with WD black drives.  My goal is to transfer data, get proper WD RE drives in the RAID array to hopefully fix the issue.

 

so here is what I have.  I'm building a minimum 14 core (expandable to 28 core) system with a norco 16bay 3u hotswap case.  I am adding a LSI 9300 HBA to the system.  I'll be installing windows 2012r2.  I'd like to try virtual box to create a VM for freenas and direct io pass through with the HBA.  I can add additional NIC if possible.

 

has anyone had any experience with this, and is it stable if so?  What is the transfer speed performance of this scenario vs loading it directly with say a 10GbE environment.  I'd like to use it long term if possible.  

 

Thanks for the help ahead of time!

[TapfererToaster]

I only ever used virtual box to quickly test something, but a quick google search leads me to believe, PCI-Passthrough is only implemented in the Linux version.

 

Quote

When running on Linux hosts, with a recent enough kernel (at least version 2.6.31) experimental host PCI devices passthrough is available.

Source: Virtualbox Manual

 

The last time I checked, HBA passthrough wasn't really working in Hyper-V either, but that might have changed - it has been i while since i checked.

 

I personally use VMware ESXi with a HBA passthrough. with it you could set up a Windows server VM and a FreeNas VM, then create a virtual-switch and connect those VMs via the vSwitch -> not bound to speed limits of physical devices when sending data to each other.

[Strikermed]

I've haven't heard of VMware.  Does it run on windows?  

 

Can you break down a virtual switch for me?  How does one do that?  I was originally planning on setting up VM's like they were independent machines with hardware dedicated to them.  As I think about this, is it possible to use at 10GbE card for two VM's? 

 

Thanks

[TapfererToaster]

VMware ESXi is a hypervisor to control VMs running on it. So, no it does not run on windows. You would install ESXi first, then create a VM and install Windows Server on it.

Depending on how you want to use your Server, this might not work out. I.E. if you want to sit in front of the server and use it as a kind of workstation. Hypervisor a typically headless systems -> they do not always have a monitor/keyboard etc connected but instead are controlled remotely over a LAN connection.

 

VMware is just one company that offers solutions like this. Microsoft got their own product, Hyper-V. But as i said in my previous post, i do not know if PCI-Passthrough works on it yet.

 

Reading your post again, I'm honestly not really sure what you are trying to do, and why?

 

If I understand you right, you got a FreeNas system running, with  WD Blacks on a  LSI 9266-8i. And in the future, you are going to build another system, add WD RE drives connected to a  LSI 9300 and add the old HBA&Drives into this new system?

 

And you want to use a FreeNas VM that got the old HBA passed through to transfer the Data from the Blacks to the RE? 

 

Sorry, if I got that wrong, but that's what i came up with..

 

23 hours ago, Strikermed said:

Can you break down a virtual switch for me?  How does one do that?  I was originally planning on setting up VM's like they were independent machines with hardware dedicated to them.

 

A vSwitch would work without any hardware. Lets say you got 2 VMs, VM-A and VM-B each got their own storage etc. and lets say your system got 2+ NICs, so they even got their own network port. 

 

Now if you want to transfer data von VM-A to VM-B you could theoretical go over the physical network port of VM-A, to a Physical switch, and then return the data to the same Server over the physical network port of VM-B. This would work pretty much out of the box, when you set up your VMs. 

But it would not really make much sense in this case. Your speed would be limited by the speed of your NICs and switch. And it does not make much sense to send the data to a switch only to send it back to the same server.

 

So hypervisor are able to create virtual switchs, virtual NICs and connect your VMs via those. 

In our exmaple, we would create an additional NIC in the VM settings of our two VMs. Then we create a virtual switch and connect those two VMs to it. And since this connection is purely virtual, it is not bound by your typical speed limits. This is of course just a broad explanation of how it would work. if you want to more info on this, i can link you to instructions of how it would be done.

 

23 hours ago, Strikermed said:

As I think about this, is it possible to use at 10GbE card for two VM's? 

If your server got a 10 GbE card in it, it could be share with every VM or you could pass it through to one single VM. (assuming it is compatible with whatever hypervisor you would chose)

But again, using this card to send data from one VM to another on the same server, would make no sense. so you would only need a 10Gb/s card if you got another server/client with another 10Gbit/s card.

[Strikermed]

22 hours ago, TapfererToaster said:

Reading your post again, I'm honestly not really sure what you are trying to do, and why?

 

If I understand you right, you got a FreeNas system running, with  WD Blacks on a  LSI 9266-8i. And in the future, you are going to build another system, add WD RE drives connected to a  LSI 9300 and add the old HBA&Drives into this new system?

 

And you want to use a FreeNas VM that got the old HBA passed through to transfer the Data from the Blacks to the RE? 

 

Sorry, if I got that wrong, but that's what i came up with..

 

 

Thanks for the reply...

 

To clear this up... right now I have my WD Black drives on the LSI 9266 8i on a windows 10 machine.  

 

I would build a server (already done by the way) running windows 2012 r2, use a virtual machine to install FreeNAS, and then use the LSI 9300 HBA to pass through to the VM and put WD RED drives on it.  

 

I would then ideally transfer all the RAID 6 info to the FreeNAS, and then come up with some other configuration and arrangement.  Essentially I would want to ensure that I can run FreeNAS on a VM, and what ever Virtual machine software I use is able to pass through the HBA so that it can access each Disk's SMART data.

 

I'm getting mixed information (probably due to different generations of software and the evolution over the years)... I've heard you can passthrough io devices, and i've heard you can't...

 

 

 

 

[Eniqmatic]

5 hours ago, Strikermed said:

Thanks for the reply...

 

To clear this up... right now I have my WD Black drives on the LSI 9266 8i on a windows 10 machine.  

 

I would build a server (already done by the way) running windows 2012 r2, use a virtual machine to install FreeNAS, and then use the LSI 9300 HBA to pass through to the VM and put WD RED drives on it.  

 

I would then ideally transfer all the RAID 6 info to the FreeNAS, and then come up with some other configuration and arrangement.  Essentially I would want to ensure that I can run FreeNAS on a VM, and what ever Virtual machine software I use is able to pass through the HBA so that it can access each Disk's SMART data.

 

I'm getting mixed information (probably due to different generations of software and the evolution over the years)... I've heard you can passthrough io devices, and i've heard you can't...

 

 

 

 

You can indeed do passthrough on ESXI, that card is also on the HCL: http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=34152&vcl=true

 

And it has the drivers you will need (if it doesn't do them at installation time) so you should be good to go if you want to go the free ESXI route!

[TapfererToaster]

10 hours ago, Strikermed said:

Essentially I would want to ensure that I can run FreeNAS on a VM, and what ever Virtual machine software I use is able to pass through the HBA so that it can access each Disk's SMART data.

This is possible, I do it myself.

 

The problem is, you're already running Windows Server on the machine. Without first deleting it, you can either install the Hyper-V role (hypervisor) for it, install Something like VMware Workstation or VirtualBox. None of those solutions would work with FreeNas 100%.

 

So here is what i would do:

(borrowed the image from the link i will post later.)

 

1. First you need to make sure your Mainboard supports VT-d. This is the option that allows PCI passthrough in the first place. If your Server has server-grade hardware, it most likely has it, but make sure to check. If you don't have it, you can scrap your plans at this point.

2. You can install and later boot ESXi from an USB-Stick.

3. After setting ESXi up, you can create your FreeNas VM. You could theoretically save the VM onto the USB-Stick which is running ESXi, but i would decide against it and rather connect a small SSD or SATA DOM to your mainboard and save the VM onto it. FreeNas needs like <10GB

4. After flashing the HBA to IT mode, pass it through to the VM in the ESXi settings. When you then boot the VM, you should see your connected drives.

5. Create a virtual switch and storage network in ESXi. connect the FreeNas VM to it.

6. Create a VM and install Windows Server 2012 R2 on it. What you then need to decide is, where do you want to save it. The most performance would be, to install it on a directly connected SSD (the one you installed FreeNas on - if its big enough). But you could also connect Storage created in the FreeNas VM to the ESXi, and then save the Windows VM there. This has the negative effect, that the Freenas VM must always be running for the windows vm to work. It's not much of a problem, if you restart the box only once or twice a year, but otherwise can be annoying.

7. you connect the Windows VM to the vSwitch.

 

This setup would allow you to run Windows Server and FreeNas at the same time. But you can only control them remotely. Since FreeNas is mostly controlled by it's webinterface and you could use remote-desktop for windows, it shouldn't really be a problem. Additionally it allows you to create all kinds of other VMs to play around with.

 

Here is a link to a site that describes how to do the points 1-5. point 6. should be easy to do after that.

https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

[Strikermed]

7 hours ago, TapfererToaster said:

This is possible, I do it myself.

 

The problem is, you're already running Windows Server on the machine. Without first deleting it, you can either install the Hyper-V role (hypervisor) for it, install Something like VMware Workstation or VirtualBox. None of those solutions would work with FreeNas 100%.

 

So here is what i would do:

(borrowed the image from the link i will post later.)

 

1. First you need to make sure your Mainboard supports VT-d. This is the option that allows PCI passthrough in the first place. If your Server has server-grade hardware, it most likely has it, but make sure to check. If you don't have it, you can scrap your plans at this point.

2. You can install and later boot ESXi from an USB-Stick.

3. After setting ESXi up, you can create your FreeNas VM. You could theoretically save the VM onto the USB-Stick which is running ESXi, but i would decide against it and rather connect a small SSD or SATA DOM to your mainboard and save the VM onto it. FreeNas needs like <10GB

4. After flashing the HBA to IT mode, pass it through to the VM in the ESXi settings. When you then boot the VM, you should see your connected drives.

5. Create a virtual switch and storage network in ESXi. connect the FreeNas VM to it.

6. Create a VM and install Windows Server 2012 R2 on it. What you then need to decide is, where do you want to save it. The most performance would be, to install it on a directly connected SSD (the one you installed FreeNas on - if its big enough). But you could also connect Storage created in the FreeNas VM to the ESXi, and then save the Windows VM there. This has the negative effect, that the Freenas VM must always be running for the windows vm to work. It's not much of a problem, if you restart the box only once or twice a year, but otherwise can be annoying.

7. you connect the Windows VM to the vSwitch.

 

This setup would allow you to run Windows Server and FreeNas at the same time. But you can only control them remotely. Since FreeNas is mostly controlled by it's webinterface and you could use remote-desktop for windows, it shouldn't really be a problem. Additionally it allows you to create all kinds of other VMs to play around with.

 

Here is a link to a site that describes how to do the points 1-5. point 6. should be easy to do after that.

https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

Thanks for the guide!  I'll try this in a week or two after I play around with Virtual box and hyperV.  I have an ASUS z10PE D16 with a single xeon 2683 at the moment... I've ensured PCI passthrough and VT-d are enabled on the board.

 

I don't think using Remote Desktop will be an issue as long as I can enable it first.  By default, Windows 2012r2 is set to disable.

 

Do you happen to know of any good videos for installing ESXi and how to use it?

 

Thanks!

[TapfererToaster]

This video seems to do a good job of explaining things:

Note that if you are following the instructions of the guide in the previous post, the guy is using a windows client - not the web gui. 

So if you want it to look like in the guide, you need to download the client. It does make some settings a bit easier, but doesn't look as fancy.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2089791

[Strikermed]

On 3/14/2017 at 8:06 PM, TapfererToaster said:

This video seems to do a good job of explaining things:

Note that if you are following the instructions of the guide in the previous post, the guy is using a windows client - not the web gui. 

So if you want it to look like in the guide, you need to download the client. It does make some settings a bit easier, but doesn't look as fancy.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2089791

Thanks!  This will definitely help!