Tips on Virtualization

[kngzeng]

Hi, the company where I work for had some parts laying around. So why not use those spare parts and build a small server.

Quote

 

Specs:

BitFenix Prodigy

Gigabyte GA-H97N-WIFI

Intel Core i3 4160

8gb DDR3 single channel

3.5 Tb of storage (on 5 different drives)

Kingston 120gb SSD

NZXT HALE82 V2 700W

(*Need to add a network card for PfSense and more RAM*)

 

 

What I want to put on this server.

-PfSense (as to upgrade our Cisco RV110W)

-FreeNAS (For backups and storage)

-Windows Server 2012 R2 (If possible setup AD, but mainly would be used for MySQL, Mail Server, and maybe to host some servers like Minecraft, TeamSpeak, CS)

 

My idea was to use Windows Server R2 and setup network, network shares, backup and everything else mainly in Windows Server but that wouldn't be an interesting project, and I was motivated by the 2 gamers and 7 gamers 1 CPU video(s) to do this project.

 

I did almost 5-6 hours of research. The best solution would be to use Unraid to do all of this, but since the company where I work for wouldn't give me more budget for this project. I needed to use a free or at least, not as expensive solution. I liked the idea to use EXSi since many recommended this but still can't afford it and free version of vSphere only allows for 1 virtual machine (or at least that is what I understood). So my next best bet is to learn to setup KVM in Linux.

 

So, I need help to setup Linux KVM, or if there is a better and still interesting way to accomplish this project. Please tell me. Any videos or guides would be appreciated. Thanks!

Also, I read in another thread, that I need a RAID card so as to FreeNAS has access to the Hard Drives and have the full benefits of ZFS. Is it true on my setup or the motherbaord I'm using isn't completely neccesary to have a RAID card.

 

[tt2468]

I would highly discourage virtualizing on the parts you are using. The core i3 only has 2 cores, so you cant get much done with it.

[tt2468]

Also, instead of using normal KVM, use proxmox. It's a free, debian based os that has a web interface so you can manage vms easier than with the shell.

[kngzeng]

Mmm, great recommendation, haven't heard about that solution before, I will do some research on it.

 

Do I still need to have a separate RAID card using Proxmox, or I can assign the drives directly to FreeNAS?

 

Thank you very much

[dalekphalm]

28 minutes ago, kngzeng said:

Mmm, great recommendation, haven't heard about that solution before, I will do some research on it.

 

Do I still need to have a separate RAID card using Proxmox, or I can assign the drives directly to FreeNAS?

 

Thank you very much

You specifically do NOT want a RAID Card, if you intend on using FreeNAS. FreeNAS needs direct low-level access to the drives, via onboard Motherboard SATA ports, or a SATA/SAS HBA card.

 

Some RAID cards can have their firmware cross flashed to "IT mode", which turns them into HBA's, but that's something that should be highly researched before buying.

 

If your motherboard has enough SATA ports for the number of drives you intend to use, then you're good to go in that respect.

[kngzeng]

Oh, then I misunderstood whatever thread I was reading of FreeNAS hehehe. 

 

Yes, I do have enough SATA ports (6 ports and 6 drives). 

 

I just finished watching tek syndicate's video on Proxmox. I did like it, tomorrow I will try it out on the machine, do some eBay research for a more powerful CPU. And see how it goes. 

 

I just have one last question for now, FreeNAS needs to be installed as a VM or as a container, the same question goes for pfSense.

 

Thanks

[dalekphalm]

Just now, kngzeng said:

Oh, then I misunderstood whatever thread I was reading of FreeNAS hehehe. 

 

Yes, I do have enough SATA ports (6 ports and 6 drives). 

 

I just finished watching tek syndicate's video on Proxmox. I did like it, tomorrow I will try it out on the machine, do some eBay research for a more powerful CPU. And see how it goes. 

 

I just have one last question for now, FreeNAS needs to be installed as a VM or as a container, the same question goes for pfSense.

 

Thanks

I'd recommend a minimum 4-core CPU for VM's - or, worst case - a dual-core w/ HT if you're only running one or two low powered VM's.

 

I'm running three VM's on a test machine (Core i7-6700 4-core w/ HT) and all three VM's run really smoothly, yet when I was running the same three VM's on a Core2Quad, they ran like shit. I also have more RAM though (16GB total on the i7 vs the original 8GB on the C2Q).

[KuJoe]

Wait a minute... you want a Server 2012 VM for MySQL? 

 

You'd be much better off running MySQL on Linux to better utilize your resources and you want to run AD on it's own VM without anything else if you plan on using it in production.

 

Lastly, you want to build a virtual router to replace your current Cisco router. Do you mean you want to replace the business router with the virtual router or is the Cisco router being used for something else. I highly discourage you experimenting with virtualization (especially virtual networking) on your company's production network unless you have really good job security, can afford to get fired, or the company doesn't use the network for business or financial stuff.

[Kadah]

How many users at this company?

 

That can be a lot of VMs for just 8GB of ram considering you want an AD on it too.

[leadeater]

3 hours ago, kngzeng said:

I liked the idea to use EXSi since many recommended this but still can't afford it and free version of vSphere only allows for 1 virtual machine (or at least that is what I understood).

ESXi is free, there are no VM limits. Only the hardware you have limits how many you can run. The paid features from VMware only come in to play with multiple hosts and requirements beyond just simply virtualizing.

 

Virtualizing FreeNAS is also not recommended, you can do it but you shouldn't. If you are going to do this you need to know more than the basic theory behind virtualization.

 

In my personal opinion if you are looking at building a system for production business use you first point of call should be ESXi. Yes I'm biased, yes other options work very well but VMware is the most widely used, best supported and the most trusted. 

[Kadah]

Just now, leadeater said:

ESXi is free, there are no VM limits. Only the hardware you have limits how many you can run. The paid features from VMware only come in to play with multiple hosts and requirements beyond just simply virtualizing.

The main issue is that ESXi likely does not support that mobo and won't be able to make a datastore. Whiteboxing ESXi is hit and miss, mostly miss.

[leadeater]

9 minutes ago, Kadah said:

The main issue is that ESXi likely does not support that mobo and won't be able to make a datastore. Whiteboxing ESXi is hit and miss, mostly miss.

True but for any virtual host using desktop parts is never recommended. If you have problems installing and running ESXi on the hardware you have you shouldn't be using it for this kind of usage at all, ESXi or anything else.

 

Totally fine thing to do for home or test lab stuff but not the smartest thing to do for business use. Refer back to my declared bias towards VMware, there is a reason why systems running ESXi are so reliable and its not all down to ESXi itself.

[Kadah]

5 minutes ago, leadeater said:

True but for any virtual host using desktop parts is never recommended. If you have problems installing and running ESXi on the hardware you have you shouldn't be using it for this kind of usage at all, ESXi or anything else.

 

Totally fine thing to do for home or test lab stuff but not the smartest thing to do for business use. Refer back to my declared biased towards VMware, there is a reason why systems running ESXi are so reliable and its not all down to ESXi itself.

I used to do it all the time, including production business use cause no budget and such at the time. It was really dumb, and caused numerous issues later on, but it worked mostly for 6+ years. Now I've can get budgets (after 6 months of requesting...) and have an entire rack for vSphere. 

 

Whitebox ESXi either works or not, pretty just has to try it and see. You can try checking 3rd party HCL lists for whiteboxing and for any driver packs that might help, but the one I used to use hasn't been updated since 4.1.

 

No idea what OP's actually intended load is yet, so no idea what would work. You can get away with a lot on small deploys. 

[kngzeng]

Updates: Got temporarily 4gb more of RAM, from a new computer I'm building, and swapping the i3 4160 with an 4170, not a big difference, but may help a little.

 

@dalekphalm Yes, I'm aware of the CPU requirements. I wouldn't have bought an i3 for this server if I had the choice, since this were hadware left from old computers or extra hardware just in case a computer broke or needed to be repaired. If I see the i3 has a hard time dealing with all the VM's I will ask for budget for a better server. I've seen many very good servers for around $200-$300 on eBay. So that would be my next option. btw, the i3 seems to have hyper threading. So thats a good thing.

 

@KuJoe Yeah, i've thought of using Linux, but since I wanted to setup AD as well, why not just use Server 2012 intead of installing another VM to the server. But I will read a bit more about that, and will maybe I will change plans. If the server is capable, I will install Linux as well and host the rest of the game servers and database servers there. Thanks for the recommendation. And in regards to the router, not really, the cisco router is being used as the primary router, but it has been giving the ghost lately, and having random issues. So this would be an upgrade to that and maybe I will use the cisco router as an access point. I do understand the risks of playing around with networking, but I think I can deal with it.

 

@Kadah maybe 30-40 devices, 15-20 users. I will upgrade to 16gb some time in the future. For now I've added 4gb more. 16 gb should be more than enough I think, or what do you recommend?

 

@leadeater Really? Oh, if thats the case I would rather go with EXSi, seems to be a more robust hypervisor. I've already downloaded most of the stuff I need to setup EXSi, but from the video I saw, I needed vCenter Server Appliance, and this is what I thought was needed to run VM and pay for. Need to do more research ehehhehe. Any guides I should look for? I will still try out ProxMox for now.

 

in General:

My only concern right now is stability. I'm aware that I'm using desktop grade hardware, and if I see that this isn't a viable way to do what I'm planning to do with this computer. I have no choice other than get some server grade hardware and use this computer maybe as another workstation for the company or anything else other than serving as a server.

 

Thanks to everyone for the help.

 

[dalekphalm]

27 minutes ago, kngzeng said:

Updates: Got temporarily 4gb more of RAM, from a new computer I'm building, and swapping the i3 4160 with an 4170, not a big difference, but may help a little.

 

@dalekphalm Yes, I'm aware of the CPU requirements. I wouldn't have bought an i3 for this server if I had the choice, since this were hadware left from old computers or extra hardware just in case a computer broke or needed to be repaired. If I see the i3 has a hard time dealing with all the VM's I will ask for budget for a better server. I've seen many very good servers for around $200-$300 on eBay. So that would be my next option. btw, the i3 seems to have hyper threading. So thats a good thing.

 

@KuJoe Yeah, i've thought of using Linux, but since I wanted to setup AD as well, why not just use Server 2012 intead of installing another VM to the server. But I will read a bit more about that, and will maybe I will change plans. If the server is capable, I will install Linux as well and host the rest of the game servers and database servers there. Thanks for the recommendation. And in regards to the router, not really, the cisco router is being used as the primary router, but it has been giving the ghost lately, and having random issues. So this would be an upgrade to that and maybe I will use the cisco router as an access point. I do understand the risks of playing around with networking, but I think I can deal with it.

 

@Kadah maybe 30-40 devices, 15-20 users. I will upgrade to 16gb some time in the future. For now I've added 4gb more. 16 gb should be more than enough I think, or what do you recommend?

 

@leadeater Really? Oh, if thats the case I would rather go with EXSi, seems to be a more robust hypervisor. I've already downloaded most of the stuff I need to setup EXSi, but from the video I saw, I needed vCenter Server Appliance, and this is what I thought was needed to run VM and pay for. Need to do more research ehehhehe. Any guides I should look for? I will still try out ProxMox for now.

 

in General:

My only concern right now is stability. I'm aware that I'm using desktop grade hardware, and if I see that this isn't a viable way to do what I'm planning to do with this computer. I have no choice other than get some server grade hardware and use this computer maybe as another workstation for the company or anything else other than serving as a server.

 

Thanks to everyone for the help.

 

One more thing to keep in mind: While you can technically run Windows Server 2012 as the Hypervisor + have AD running at the same time, it's HIGHLY recommended to make sure that the AD is running completely separate as it's own VM. This will obviously be the case if you choose to run ESXi or ProxMox, etc.

 

Another thing, vCenter is the interface/management app/guest for ESXi - I don't think you have to run it, but it makes administering the server a million times easier.

 

We're actually about to start a VM server project at work: ~$100,000 project. 2 or 3 nodes (blade servers) in a cluster, ESXi load balancing between them, running anywhere from 4 to 12 VM's. We're virtualizing almost our entire server infrastructure.

[Smite]

45 minutes ago, dalekphalm said:

One more thing to keep in mind: While you can technically run Windows Server 2012 as the Hypervisor + have AD running at the same time, it's HIGHLY recommended to make sure that the AD is running completely separate as it's own VM.

Let me bold the rest of that for the OP and reiterate it again. Run AD on its own VM, 

You are also trying to combine three things that inherently don't like to work well with one another. 

Pfsense - I always recommend its own box, many reasons behind this. 

Freenas - great for nas, not so great at virtualization. The point of freenas is to have a great NAS OS running for your Network storage. 

Windows Server 2012 - has built in hypervisor, and you don't need more licenses to run up to two instances of server 2012...

 

[kngzeng]

Thanks @dalekphalm and @Smite for the advice. 

 

I will test each one separately, install Pfsense, get to know the software, then install FreeNAS, and then try all of them running under ESXi. If performance is not great or is not stable at all, will go back to the original plan, which is to install Windows Server 2012 R2, and use the features available on Windows Server (DHCP, AD, SMB, etc.). 

 

If so, my undestanding, is that I should run Windows Server, use Hyper-V, install an instance of Windows Server and run AD from there, right? Are the other features, for example Disk Pools, Shares and else should be on a virtual instance under Hyper-V or could they be running side by side with Hyper-V?

 

[dalekphalm]

48 minutes ago, Smite said:

Let me bold the rest of that for the OP and reiterate it again. Run AD on its own VM, 

You are also trying to combine three things that inherently don't like to work well with one another. 

Pfsense - I always recommend its own box, many reasons behind this. 

Freenas - great for nas, not so great at virtualization. The point of freenas is to have a great NAS OS running for your Network storage. 

Windows Server 2012 - has built in hypervisor, and you don't need more licenses to run up to two instances of server 2012...

 

Personally, I would either: Run FreeNAS as it's own box - or scrap FreeNAS all together and run something like UnRAID or simply Linux or Windows Server, for the NAS itself.

 

While it's theoretically possible to virtualize FreeNAS, the devs don't support or recommend it, because things get extremely complicated and tricky, and are prone to weird issues and incompatibilities.

[TapfererToaster]

As far as i know, the i3-4170 does not support VT-d, is that right?

 

If not, i see a big problem with running ESXi and a FreeNas VM on the system.

To make sure the data FreeNas writes on your HDDs is correct, you MUST use Passthrough to pass the Raid/SAS controller connected to your HDDs to the VM.

The Sata/SAS controller can be the one on your mainboard or a seperate one, a popular choice is a LSI 9211-8i card, flashed to IT mode, that comes in different OEM brands.

 

That leaves you with some problems:

Even IF the mainboard and the CPU support VT-d, you typically need 2 different SATA/SAS-controller.

One on which the FreeNas, PfSense and windows VMs are installed (most likely on the SSDs), and one controller that you can passthrough to the Freenas VM that is connected to the Storage HDDs.

 

There are other ways to do it, I.E:

install FreeNas on some kind of USB device and Passthrough the mainboard controller with all SSD+HDDs and share those drives in FreeNas with the ESXi host as Datastorage.

That way, you can install the other VMs on the newly added datastore. But since i never tried installing a FreeNas VM on a  USB device, i can't tell you how stable it would run.

 

Some Mainboards already got 2 different controller on it (most of the times they are a different color). I think i got an old Gigabyte with one controller with 2 Sata-ports and another with 4-ports.

 

[legopc]

18 hours ago, kngzeng said:

I liked the idea to use EXSi since many recommended this but still can't afford it and free version of vSphere only allows for 1 virtual machine (or at least that is what I understood).

 

 

 

Nope, you can have as many vm's as you want with the free version of ESXI. And please do look into using ESXI seeing as it is a great hypervisor.

[Kadah]

14 hours ago, dalekphalm said:

One more thing to keep in mind: While you can technically run Windows Server 2012 as the Hypervisor + have AD running at the same time, it's HIGHLY recommended to make sure that the AD is running completely separate as it's own VM. This will obviously be the case if you choose to run ESXi or ProxMox, etc.

 

13 hours ago, Smite said:

Let me bold the rest of that for the OP and reiterate it again. Run AD on its own VM, 

I'm going to do the same too because I found this out the bad, expensive way. Despite what MS says about R2, AD cannot coexist with any other major roles petty much other than file service. And its it recommended to have the hypervisor host running only Hyper-V role, which makes means more resource waste vs ESXi/vSphere.

You can run two VM instances of Server 2012 R2 with a single license on a single host regardless of the hypervisor.

I've been looking at standalone free ESXi vs Hyper-V for a single host with two 2012 R2 VMs and 1-2 Linux VMs. I could not find any upsides to going with Hyper-V for this, either it was going to be the same or worse with Hyper-V.

 

10 hours ago, TapfererToaster said:

As far as i know, the i3-4170 does not support VT-d, is that right?

The i3-4170 supports VT-x.

 

14 hours ago, kngzeng said:

16 gb should be more than enough I think, or what do you recommend?

16GB would be good.

[leadeater]

15 hours ago, dalekphalm said:

One more thing to keep in mind: While you can technically run Windows Server 2012 as the Hypervisor + have AD running at the same time, it's HIGHLY recommended to make sure that the AD is running completely separate as it's own VM. This will obviously be the case if you choose to run ESXi or ProxMox, etc.

 

Another thing, vCenter is the interface/management app/guest for ESXi - I don't think you have to run it, but it makes administering the server a million times easier.

 

We're actually about to start a VM server project at work: ~$100,000 project. 2 or 3 nodes (blade servers) in a cluster, ESXi load balancing between them, running anywhere from 4 to 12 VM's. We're virtualizing almost our entire server infrastructure.

Personally I wouldn't go with blade servers, we use 55 (ish) HP DL360 Gen 9 2x E5-2690v3 386GB RAM with onboard dual 10Gb with another dual 10Gb network card and use NFS storage. Each host depending on site either has an average of 20 VMs or 40 VMs, we have around 1000 so it averages out to about 18 per host but half are DR standby so effectively double that. 

 

On the face of it blade severs might seem like a good choice but have limited expandability options and the single chassis in your proposed deployment is a single point of failure (backplane and power).

 

Not sure what storage you are also planning but I would invite a Nutanix sales engineer in to get a brief on what they can offer. If both storage and servers are going to be new this may actually be your cheapest option and is an excellent product. We use Nutanix for our development VM cluster and for one of the DR sites.

 

Also 4-12 VMs is on the low side so 3 nodes would make for a very expensive $/VM figure, but once you start virtualizing VM counts typical jump up very quickly so getting 3 now might be a good choice. 

[leadeater]

16 hours ago, kngzeng said:

@leadeater Really? Oh, if thats the case I would rather go with EXSi, seems to be a more robust hypervisor. I've already downloaded most of the stuff I need to setup EXSi, but from the video I saw, I needed vCenter Server Appliance, and this is what I thought was needed to run VM and pay for. Need to do more research ehehhehe. Any guides I should look for? I will still try out ProxMox for now.

 

vCenter is only required for managing multiple physical servers hosting VMs, it is not free either. The vSphere client can connect directly to the server or to vCenter, it is the same application and looks exactly the same connected to the host or vCenter. There are features the can only be used and will only show when connected to vCenter.

 

If you have multiple servers even then you still do not need vCenter, you can manage them all separately. It all comes down to your requirements.

 

With the type of hardware you are using you might have better experience with proxmox or KVM etc. ESXi really wants to be installed on server hardware not desktop. Give it a try anyway, be good experience even if you dont use it. Just install ESXi on a USB drive that way you can just take it out once your done and have the option to just plug it back in at a later time and boot straight back in to it.

[KuJoe]

23 minutes ago, leadeater said:

Personally I wouldn't go with blade servers, we use 55 (ish) HP DL360 Gen 9 2x E5-2690v3 386GB RAM with onboard dual 10Gb with another dual 10Gb network card and use NFS storage. Each host depending on site either has an average of 20 VMs or 40 VMs, we have around 1000 so it averages out to about 18 per host but half are DR standby so effectively double that. 

 

On the face of it blade severs might seem like a good choice but have limited expandability options and the single chassis in your proposed deployment is a single point of failure (backplane and power).

 

Not sure what storage you are also planning but I would invite a Nutanix sales engineer in to get a brief on what they can offer. If both storage and servers are going to be new this may actually be your cheapest option and is an excellent product. We use Nutanix for our development VM cluster and for one of the DR sites.

 

Also 4-12 VMs is on the low side so 3 nodes would make for a very expensive $/VM figure, but once you start virtualizing VM counts typical jump up very quickly so getting 3 now might be a good choice. 

I'm going to disagree here. Blades are awesome for ESXi hosts in terms of deployment, management, and capacity per cabinet. I'm not a big fan of HP hardware anymore (we're phasing our the DL380/580s and BL460s for Cisco blades and only using rackmount servers for management hosts), but they do many some solid blades and the chassis aren't too bad although a bit power hungry. The single point of failure is a pain, but mulitple chassis will eliminate that and if you leave some slots open on other chassis you can easily move the blades to a working chassis if something fails on another chassis (and don't put all the hosts for one cluster in a single chassis).

 

As for storage, we've started phasing out EMC for Pure Storage because they out perform EMC's XtremeIO in every test, are extremely easy to manage and setup, the price per TB is much cheaper, and takes up only a quarter cabinet compared to the cabinet sized SANs EMC offers.

[leadeater]

3 minutes ago, KuJoe said:

I'm going to disagree here. Blades are awesome for ESXi hosts in terms of deployment, management, and capacity per cabinet. I'm not a big fan of HP hardware anymore (we're phasing our the DL380/580s and BL460s for Cisco blades and only using rackmount servers for management hosts), but they do many some solid blades and the chassis aren't too bad although a bit power hungry. The single point of failure is a pain, but mulitple chassis will eliminate that and if you leave some slots open on other chassis you can easily move the blades to a working chassis if something fails on another chassis (and don't put all the hosts for one cluster in a single chassis).

 

As for storage, we've started phasing out EMC for Pure Storage because they out perform EMC's XtremeIO in every test, are extremely easy to manage and setup, the price per TB is much cheaper, and takes up only a quarter cabinet compared to the cabinet sized SANs EMC offers.

Yea I'm not saying don't use blade, just not in this case where it will only be one chassis.