Dumb FreeNAS question

[AbsoluteFool]

Ok i have a question about running virtualized FreeNAS. I know virtualized is not supported and shouldn't be done.

But i wonder if some one of you have a bit more experience with the situation that i'm thinking about than me. So this is my question.

 

If i wanted to use a workstation like Ubuntu Desktop and run FreeNAS from Virtualbox. I would then create each virtual disk from each their physical disk. (Without any RAID controller of course)

How would virtual disks have negative effect on FreeNAS? As they are not used to anything else then Virtual disks for the FreeNAS VM?

 

As far from my knowledge FreeNAS does not play nice with virtual disks, because of the way the drives are presented to FreeNAS if this was from a RAID controller perspective. Does the same issues apply without a RAID controller? Or am i really just chasing my own tail? lol

[Electronics Wizardy]

12 minutes ago, AbsoluteFool said:

 

If i wanted to use a workstation like Ubuntu Desktop and run FreeNAS from Virtualbox. I would then create each virtual disk from each their physical disk.

why would you do this? Just install zfs on ubuntu and make your array on the host. You have the exact same storage system and can easily make shares.  I don't see any reason for a freenas vm here.

[AbsoluteFool]

5 minutes ago, Electronics Wizardy said:

why would you do this? Just install zfs on ubuntu and make your array on the host. You have the exact same storage system and can easily make shares.  I don't see any reason for a freenas vm here.

Yes, i know. That's probably what i'm going to do. Was just something that came to my mind, however dumb that might sound.

[Razor Blade]

I do have FreeNAS in an ESXi VM. FreeNAS will work with virtual disks but there noted issues and risks by doing this. One example would be if there is a failure and you end up reinstalling FreeNAS, normally you can just restore the data by importing the disks no problem. With virtual disks that may not be so. Also with virtual disks you also may not be able to take advantage of FreeNAS's ability to monitor them with S.M.A.R.T tests which can help tell if a disk is healthy or if it could be starting to have problems.

 

The best way to go about it is to pass through an HBA controller to FreeNAS and let it do it's thing. That is how I do it. FreeNAS knows no difference. The drives are presented to it as if it was on a bare metal box.

[dalekphalm]

11 hours ago, AbsoluteFool said:

Ok i have a question about running virtualized FreeNAS. I know virtualized is not supported and shouldn't be done.

There's actually no problem virtualizing FreeNAS. The main reason why it was "unsupported" is because people would just try to throw it into VirtualBox or VMWare Player without any Hardware Passthrough. FreeNAS (Well, ZFS specifically) requires direct access to your HDD's. If you can give it that, it doesn't care whether it's virtualized or not.

11 hours ago, AbsoluteFool said:

But i wonder if some one of you have a bit more experience with the situation that i'm thinking about than me. So this is my question.

I've been running FreeNAS in a VM under ESXi for like a year - it runs perfectly.

11 hours ago, AbsoluteFool said:

If i wanted to use a workstation like Ubuntu Desktop and run FreeNAS from Virtualbox. I would then create each virtual disk from each their physical disk. (Without any RAID controller of course)

Do. Not. Do. This.

 

If you are going to use FreeNAS, give FreeNAS direct access to the drives. If you want to do this with a VM, that means using a VM (and hardware) that supports PCIPassthrough - I know ESXi supports this, as does unRAID - others likely do as well.

11 hours ago, AbsoluteFool said:

How would virtual disks have negative effect on FreeNAS? As they are not used to anything else then Virtual disks for the FreeNAS VM?

Basically, you entirely negate the very reason for using FreeNAS - ZFS. ZFS relies on having direct and full access to the HDD's - in particular SMART data and SMART commands. If you use virtual disks, ZFS no longer has that control.

This means ZFS can not guarantee or even really have any idea of whether your drives are good or not.

11 hours ago, AbsoluteFool said:

As far from my knowledge FreeNAS does not play nice with virtual disks, because of the way the drives are presented to FreeNAS if this was from a RAID controller perspective. Does the same issues apply without a RAID controller? Or am i really just chasing my own tail? lol

You're correct that FreeNAS does not play well with virtual disks. As noted above, it eliminates some of the best benefits of ZFS.

 

What you want to do, is to get an HBA Card (not a RAID Card - although you can use a RAID Card in "IT" mode, which basically turns it into an HBA). Tell your Hypervisor to pass the HBA card directly to the FreeNAS VM, and then connect all your drives to said HBA Card.

 

When you boot into the FreeNAS VM, it'll have direct access to the drives, no different from a "bare metal" FreeNAS installation.

[AbsoluteFool]

5 hours ago, dalekphalm said:

There's actually no problem virtualizing FreeNAS. The main reason why it was "unsupported" is because people would just try to throw it into VirtualBox or VMWare Player without any Hardware Passthrough. FreeNAS (Well, ZFS specifically) requires direct access to your HDD's. If you can give it that, it doesn't care whether it's virtualized or not.

I've been running FreeNAS in a VM under ESXi for like a year - it runs perfectly.

Do. Not. Do. This.

 

If you are going to use FreeNAS, give FreeNAS direct access to the drives. If you want to do this with a VM, that means using a VM (and hardware) that supports PCIPassthrough - I know ESXi supports this, as does unRAID - others likely do as well.

Basically, you entirely negate the very reason for using FreeNAS - ZFS. ZFS relies on having direct and full access to the HDD's - in particular SMART data and SMART commands. If you use virtual disks, ZFS no longer has that control.

This means ZFS can not guarantee or even really have any idea of whether your drives are good or not.

You're correct that FreeNAS does not play well with virtual disks. As noted above, it eliminates some of the best benefits of ZFS.

 

What you want to do, is to get an HBA Card (not a RAID Card - although you can use a RAID Card in "IT" mode, which basically turns it into an HBA). Tell your Hypervisor to pass the HBA card directly to the FreeNAS VM, and then connect all your drives to said HBA Card.

 

When you boot into the FreeNAS VM, it'll have direct access to the drives, no different from a "bare metal" FreeNAS installation.

Aha, Thank you my friend. I was just chasing my own tail yet again, but that is the reason why i asked this dumb question haha. Cheers 

[dalekphalm]

1 minute ago, AbsoluteFool said:

Aha, Thank you my friend. I was just chasing my own tail yet again, but that is the reason why i asked this dumb question haha. Cheers 

No worries - feel free to ask any questions you like.

 

Frankly there are a lot of (incorrect) myths about FreeNAS and ZFS - for example, the 1GB of RAM per 1TB of storage - totally incorrect (Unless you're using Dedupe, which 99% of FreeNAS users aren't doing, and Dedupe on ZFS sucks anyway).

 

So the "don't virtualize FreeNAS" thing also kind of follows the same trend. You can totally do it as long as you design your setup correctly.

[jde3]

ZFS can run on a virtual disk. There is nothing.... really bad about that other than it might be inefficient as Virtual Disks use different (worse) storage layouts and formats. People run ZFS on Luks and Geli volumes all the time and thats basically a container disk. Make sure that you have (host IO) caching off or you will fuck your disk up horribly bad. ZFS has to know that writes have completed because it's transnational and it has it's own cache. It can not fsck itself back to a working state.

 

And yes you can add a ZFS volume to almost any Linux distro. Even SuSE that is a huge (the only?) btrfs supporter has a ZFS module you can just install. You can also put Linux on ZFS root, but that is a little more challenging. (Gentoo is probably the easiest to do that with)

 

You can also do it with a physical and that is fine as well so long as all the other OS's know to keep their filthy hands off it. I have a living situation where I only have one computer at the moment. (temporary) but I need different OS's due to my work. I have a dual boot Windows/Linux setup with FreeBSD on a physical disk attached to Virtualbox. This is so the guest (FreeBSD) can run irregardless of the OS the host running. In theory I could boot that FreeBSD volume directly and launch Linux into the VM, Windows may choke tho.. Why crummy old Virtualbox? because it's cross platform.

 

Did you know... That you can run X in a FreeBSD Jail? A jail can access your video card drivers just fine something Docker can't do. So in theory, you could install FreeBSD, or GhostBSD into a FreeNAS jail and have "virtualisation" and full hardware support without a hypervisor. When I get setup properly I'll probably make myself a FreeBSD workstation and keep most of base system default, and just Jail out every part of it I want to use.. so I'll create a Jailed desktop and various Jailed servers. Why? Because it makes them disposable.. if I want a KDE desktop I can install that stuff and if I change my mind I can nuke it and install XFCE separately. No cruft, no unnecessary packages.

[AbsoluteFool]

21 hours ago, jde3 said:

ZFS can run on a virtual disk. There is nothing.... really bad about that other than it might be inefficient as Virtual Disks use different (worse) storage layouts and formats. People run ZFS on Luks and Geli volumes all the time and thats basically a container disk. Make sure that you have (host IO) caching off or you will fuck your disk up horribly bad. ZFS has to know that writes have completed because it's transnational and it has it's own cache. It can not fsck itself back to a working state.

 

And yes you can add a ZFS volume to almost any Linux distro. Even SuSE that is a huge (the only?) btrfs supporter has a ZFS module you can just install. You can also put Linux on ZFS root, but that is a little more challenging. (Gentoo is probably the easiest to do that with)

 

You can also do it with a physical and that is fine as well so long as all the other OS's know to keep their filthy hands off it. I have a living situation where I only have one computer at the moment. (temporary) but I need different OS's due to my work. I have a dual boot Windows/Linux setup with FreeBSD on a physical disk attached to Virtualbox. This is so the guest (FreeBSD) can run irregardless of the OS the host running. In theory I could boot that FreeBSD volume directly and launch Linux into the VM, Windows may choke tho.. Why crummy old Virtualbox? because it's cross platform.

 

Did you know... That you can run X in a FreeBSD Jail? A jail can access your video card drivers just fine something Docker can't do. So in theory, you could install FreeBSD, or GhostBSD into a FreeNAS jail and have "virtualisation" and full hardware support without a hypervisor. When I get setup properly I'll probably make myself a FreeBSD workstation and keep most of base system default, and just Jail out every part of it I want to use.. so I'll create a Jailed desktop and various Jailed servers. Why? Because it makes them disposable.. if I want a KDE desktop I can install that stuff and if I change my mind I can nuke it and install XFCE separately. No cruft, no unnecessary packages.

I do like Virtualbox for some reason. Don't get me wrong i'd rather use ESXI but money can't always reach everything you'll ever want. I'm totally fine using Virtualbox. 

 

I can see that your situation is kinda the same as mine. I do have alot of servers but the power prices are rasing so insanely so i'd rather try and virtualize as much as possible as my needs are not really big. I do also have better backup than most people so i don't think dead drives or really doing big mistakes will make me lose any data. However, i do believe clearifications are very important, so if i actually do something dumb i will be aware of it.

 

I'll defantly look into jails on FreeBSD. I don't have any experience with FreeBSD but as i'm quite known with Linux i'm sure it can't be too hard.

[Razor Blade]

54 minutes ago, AbsoluteFool said:

I do like Virtualbox for some reason. Don't get me wrong i'd rather use ESXI but money can't always reach everything you'll ever want. I'm totally fine using Virtualbox. 

 

I can see that your situation is kinda the same as mine. I do have alot of servers but the power prices are rasing so insanely so i'd rather try and virtualize as much as possible as my needs are not really big. I do also have better backup than most people so i don't think dead drives or really doing big mistakes will make me lose any data. However, i do believe clearifications are very important, so if i actually do something dumb i will be aware of it.

 

I'll defantly look into jails on FreeBSD. I don't have any experience with FreeBSD but as i'm quite known with Linux i'm sure it can't be too hard.

I know next to nothing about FreeBSD but I do use PFsense and FreeNAS (both I believe are FreeBSD based). They just...work...so on the occasion something doesn't work or I don't know how to set something up, I just Google it. If it is an odd problem that doesn't seem to have a solution or much information about it online, I post about it on here and put it in my signature to try to help other people out.

 

Also ESXI is free...you can sign up as a student or a business and you'll get the bare metal hypervisor with all basic functions unlocked. It is Vsphere and the other premium products you pay out the butt for... I am playing around with other hypervisors right now because there doesn't seem to be a simple way to migrate, clone, or backup VMs in ESXI right now without the other premium tools (that and the fact that ESXI is quickly discontinuing support for older hardware). As a side note...I'm not below paying for premium tools, it's just paying $2,000+ for enterprise software suites is way too steep for what is essentially a hobby...seems to be why a lot of people end up using trial versions of many server OS like windows server. Which is fine to install and mess with but I can't in my right judgement deploy it to run my house and have to redo everything every 6 months when the trial expires.

 

I know that I could probably make things more efficient such as setting up many services I use into a server OS like ubnuntu, cent, etc. instead of using a tier 1 hypervisor and VMs to run things but separating things is easier (for me) to troubleshoot them. I just don't want one change to one aspect to affect another aspect if that makes sense. I tend to find what works and find a way to combine it with the other things that worked. That is why a hypervisor is perfect for my use case. I can run all those separate things on one box which saves me about 130 watts of power compared to bare metal and will also WAY better utilize the power of my server instead of it just idling all day.

[jde3]

2 hours ago, Razor Blade said:

I know next to nothing about FreeBSD but I do use PFsense and FreeNAS (both I believe are FreeBSD based). They just...work...so on the occasion something doesn't work or I don't know how to set something up, I just Google it. If it is an odd problem that doesn't seem to have a solution or much information about it online, I post about it on here and put it in my signature to try to help other people out.

 

Also ESXI is free...you can sign up as a student or a business and you'll get the bare metal hypervisor with all basic functions unlocked. It is Vsphere and the other premium products you pay out the butt for... I am playing around with other hypervisors right now because there doesn't seem to be a simple way to migrate, clone, or backup VMs in ESXI right now without the other premium tools (that and the fact that ESXI is quickly discontinuing support for older hardware). As a side note...I'm not below paying for premium tools, it's just paying $2,000+ for enterprise software suites is way too steep for what is essentially a hobby...seems to be why a lot of people end up using trial versions of many server OS like windows server. Which is fine to install and mess with but I can't in my right judgement deploy it to run my house and have to redo everything every 6 months when the trial expires.

 

I know that I could probably make things more efficient such as setting up many services I use into a server OS like ubnuntu, cent, etc. instead of using a tier 1 hypervisor and VMs to run things but separating things is easier (for me) to troubleshoot them. I just don't want one change to one aspect to affect another aspect if that makes sense. I tend to find what works and find a way to combine it with the other things that worked. That is why a hypervisor is perfect for my use case. I can run all those separate things on one box which saves me about 130 watts of power compared to bare metal and will also WAY better utilize the power of my server instead of it just idling all day.

"both I believe are FreeBSD based" - Yes, they are.

 

Try bhyve maybe. It works with ZFS ZVOLS so you can just snapshot it. Take a look at http://chyves.org

 

Bhyve is a little new.. so not everything is quite perfect.. but at least they have the storage part figured out.

 

[Razor Blade]

5 minutes ago, jde3 said:

"both I believe are FreeBSD based" - Yes, they are.

 

Try bhyve maybe. It works with ZFS ZVOLS so you can just snapshot it. Take a look at http://chyves.org

 

Bhyve is a little new.. so not everything is quite perfect.. but at least they have the storage part figured out.

 

The only experience I had with bhyve is messing around in FreeNAS. While I could get a VM running no problem, I had a bit of trouble passing through hardware and configuring the virtual network the way I wanted it. Seemed like there were some people that post online having trouble with hardware compatibility though it has certainly come a long way. Not sure how well FreeNAS really represents bhyve as a whole but seems like bhyve it certainly is a good option.

[jde3]

41 minutes ago, Razor Blade said:

The only experience I had with bhyve is messing around in FreeNAS. While I could get a VM running no problem, I had a bit of trouble passing through hardware and configuring the virtual network the way I wanted it. Seemed like there were some people that post online having trouble with hardware compatibility though it has certainly come a long way. Not sure how well FreeNAS really represents bhyve as a whole but seems like bhyve it certainly is a good option.

FreeBSD is getting a new virtual networking implementation for data centres.. unsure about their progress here or if this is going to be accepted mainline but it looks good. VNET is what we currently have and it has some problems.

 

 

This is 3 times faster than Amazon Ec2 and other Linux implementations.

[AbsoluteFool]

4 hours ago, Razor Blade said:

I know next to nothing about FreeBSD but I do use PFsense and FreeNAS (both I believe are FreeBSD based). They just...work...so on the occasion something doesn't work or I don't know how to set something up, I just Google it. If it is an odd problem that doesn't seem to have a solution or much information about it online, I post about it on here and put it in my signature to try to help other people out.

 

Also ESXI is free...you can sign up as a student or a business and you'll get the bare metal hypervisor with all basic functions unlocked. It is Vsphere and the other premium products you pay out the butt for... I am playing around with other hypervisors right now because there doesn't seem to be a simple way to migrate, clone, or backup VMs in ESXI right now without the other premium tools (that and the fact that ESXI is quickly discontinuing support for older hardware). As a side note...I'm not below paying for premium tools, it's just paying $2,000+ for enterprise software suites is way too steep for what is essentially a hobby...seems to be why a lot of people end up using trial versions of many server OS like windows server. Which is fine to install and mess with but I can't in my right judgement deploy it to run my house and have to redo everything every 6 months when the trial expires.

 

I know that I could probably make things more efficient such as setting up many services I use into a server OS like ubnuntu, cent, etc. instead of using a tier 1 hypervisor and VMs to run things but separating things is easier (for me) to troubleshoot them. I just don't want one change to one aspect to affect another aspect if that makes sense. I tend to find what works and find a way to combine it with the other things that worked. That is why a hypervisor is perfect for my use case. I can run all those separate things on one box which saves me about 130 watts of power compared to bare metal and will also WAY better utilize the power of my server instead of it just idling all day.

I already use ESXI on all of my servers. The problem is that i cannot have 100+ servers running 24/7. And i'm trying to sort out the bare minimume of my needs into one or two servers that will work optimal for my needs, as i can always power on another for the time i need something more.

 

I'm also quite curious on XCP-NG but haven't come around to try it.

Proxmox is also a nice hypervisor people say. But i just get so damn turned off by their UI that i don't even bother using it. lol bad excuse haha