ESXI Home Lab Considerations

DogKnight · April 10, 2019

Hello all,

When digging through some files I can across a small guide that I put together several years ago for those considering building their own home ESXI lab. It is far from comprehensive and mainly focuses on hardware considerations. However as this topic seems to come up fairly regularly, I thought it could be useful to some people.

Please remember this is a few years old so some of the information won't be ideal as of today, but it should still prove as a good starting point. If you find it helpful, let me know.

Enjoy!

Quote

Today I am going to be talking about putting together hardware for the purpose of running VMware’s ESXi hypervisor in your own home. There is a multitude of different paths that can be taken, and which suits you best as an individual will depend on what you want to use the lab for. So before I go into details, lets briefly talk about why somebody would want to setup their own ESXi Lab.

Reasons for setting up a home ESXi Lab often fall into two main categories.

The first is Education. People who fit into this category are those who want to learn more about the system. Whether it is to complete an official certification or because they use the systems at their place of work and want an environment where they can try new things without any real-world consequences. Home Labs can provide an excellent learning environment where you can get your hands dirty and try out new systems or tools.

The second is because you have a love for Tech. Whether you want to centralize the management of your home network, setup additional services, or just enjoying tinkering with something new. The fact that the ESXi system and its management software vSphere are free; is often enough reason for some people to have a go at this type of setup.

When it comes to setting up a home lab, there is always a set of limitations in place. These limitations can affect the end result in many different ways. So lets talk about limitations that individuals may face when setting up their own home lab.

1)      Money

Everyone has a budget on how much he or she is prepared to spend on a home lab. Whether it is $500 or $5000. Budgets exist for all of us. And even if funds are aplenty, there is your own personal limit on how much you would want to dedicate to this project. You can easily setup ESXi on some old server hardware that you saved from the trash, but for a multitude of reasons, this may not be a good option.

2)      Environment

ESXi servers are not like home computers that you may shut down every day when you are done with them. They are usually always left turned on. This will depend on what you are using it for, however 24/7 operation is the norm. If you are using either new or old server grade equipment, there are several drawbacks.

-          Electricity – Server grade hardware often uses large amount of electricity. Regularly having redundant power supplies and cooling components, they will use a lot more electricity than a standard desktop computer. This is important if you do not want a large electricity bill. If the server is running 24/7, power efficiency is definitely something you want to consider.

-          Heat – Whichever hardware you choose, if it is running 24/7 it will generate a fair amount of heat. It is important to make sure during the warmer months that you can provide adequate cooling for the equipment. This is especially true if you plan on having more than one server.

-          Location – Multiple servers can take up a lot of room. Decide early on whether you would like to buy rack mountable equipment, or more conventional hardware.

-          Noise – Server hardware specifically can be very noisy and definitely not something you want next to a bedroom. Plan your equipment based around where it will be located.

All of these aspects combined can lead to problems if not planned well. A server that keeps you awake at night, turns a room into a furnace and adds hundreds of dollars to your electricity bill is not going to be popular with anyone else living with you.

3)      Time

Even with everything else taken into consideration, the process will take time. Building your own PC for the purpose can be fulfilling, help you get exactly the resources you need, and possible save you a few dollars. But the research time it would take to perfect all of these can be substantial, especially if this is your first attempt. Purchasing a brand name server can save a lot of time. And if it is on VMware’s certified list of equipment, can have the added benefit of having Technical Support available.

Now that we have covered some of the limitations to consider when buying equipment for your home lab, lets briefly talk about other considerations outside of the primary computer or server hardware.

1)      Networking

VMware’s ESXi requires the host to have at least gigabit Ethernet available. I would recommend these days that the whole network be at gigabit speeds. If this lab is to provide services, consider whether or not a smart switch that can handle IP based routing is required.

2)      Licensing

It is important to keep in mind that VMware’s free version of ESXi and vSphere, does not allow full functionality. If you want to use features such as High Availability, DRS or Fault Tolerance, without a purchase, you’ll usually be limited to a 60-day trial. So depending on your plans, you may need to factor in the cost of purchasing a license. If your place of work is a VMware partner, there can usually be some discounted options available.

All right, now lets get down to the hardware, which will run the ESXi system and your VM’s.

There are multiple requirements for ESXi that pertain to hardware specifications and functionality. I will run through each of these and elaborate a bit more on their requirements.

-          The CPU

First of all, the CPU needs to have two physical cores at a minimum and needs to be a 64-bit processor from 2006 or newer. This requirement is fairly easy to meet. Almost all current processors contain multiple CPU cores. With desktop CPU’s ranging between 2 and 8 cores. And server grade CPU’s ranging from 2 to 18 cores. It is also worth mentioning that ESXi can take advantage of features such as Hyper Threading. This allows each CPU core to process two individual instructions simultaneously, allowing ESXi to see a two-core processor as having 4 available cores.

In addition to this, there are several CPU features required for ESXi.

-          The NX/XD bit needs to be enabled in the CPU. These abbreviations stand for Never Execute (NX) in AMD processors or Execute Disable (XD) in Intel processors. This ability prevents applications from one VM, accessing memory space used by other VM’s.

-          Intel VT-X or AMD RVI is needed in order to host 64-bit operating systems. Often this functionality needs to be enabled in the BIOS of older motherboards too. This sometimes requires a firmware update as well. It is important to make sure the motherboard supports this functionality, as the CPU having the capabilities alone will not allow it to function.

-          Depending on what you plan on using the VM’s for, you may also want to consider if your processor and motherboard support Intel VT-D or AMD IOMMU. These allow for a functionality called DirectPath I/O. This allows you to directly connect specific hardware components to a single VM. This can be useful if for example you wanted to do gaming on a specific VM. By installing a second graphic card on your host, you could connect this directly to a VM for gaming purposes.

-          RAM

-          VMware requires a minimum of 4gb of RAM for ESXi. In reality 8gb of RAM is a more realistic minimum if you want to host any VM’s in your environment. Now VMware loves RAM. The more RAM you can afford for your ESXi host the better and this should take precedence over CPU clock speed. All processors and motherboards are limited to how much RAM they can support, so plan ahead. Most modern desktop grade processors will support 16gb of RAM, with performance-oriented models supporting 32gb or even 64gb of RAM.

-          Server class processors will support a lot more than this, reaching up into the hundreds of gigabytes. However server grade hardware often makes use of ECC RAM. ECC stands for Error Correcting Code. This memory can tolerate single-bit errors by making use of extra memory bits included in the RAM modules. ECC RAM is therefore much more stable and reliable than non-ECC RAM used in consumer grade hardware. The downside of this, is that ECC RAM is much more expensive. So if you are to look at server grade processors and motherboards, allow extra money in your budget than you normally would for RAM.

-          NIC

-          ESXi requires a minimum of gigabit (1000 base) Ethernet in order to even install. Ideally having two-gigabit Ethernet interfaces would allow you more flexibility in the services you can run on your host. Be aware that ESXi is particularly fussy with which NIC’s it supports. A lot of on-board Ethernet connections in consumer grade products are not supported. For this reason, purchase of an additional NIC with a supported chipset may be required. It is recommended that you check VMware’s compatibility guide to make sure the hardware you plan on purchasing is supported.

-          You can customize the ESXi installer to include additional drivers. VMware specifically provide a tool for this. However this of course relies on the correct drivers being available at all. Often this works best if a specific NIC chipset was supported previously by ESXi, but has since had support removed.

-          Storage

-          For the ESXi system itself, not much storage space is required. An 8gb USB stick would suffice for the ESXi install and leave room for the scratch partition. If you are planning on running the OS from a USB stick, I would recommend a good quality 16gb stick. Extra flash cells in larger capacity drives can extend their life. And better quality drives will last longer.

-          You can install ESXi on local SATA or SAS drives. Ideally, having an SSD for the OS and scratch volume would be best for performance reasons. If you do use a local mechanical drive for the OS or as a data store, that will likely become the performance bottleneck in your environment. Also, as with any system, this leaves a single point of failure, which is far from ideal.

-          A RAID setup is always highly recommended. Bare in mind that almost all motherboards only offer software based RAID. Even if enabled in the BIOS, as it is a software based RAID, ESXi will not recognise it.

-          If you want to do a local raid, you will need to look at a dedicated PCI-E RAID controller. These can be quite pricey though.

-          Another option is to create a VM on your host and use an option like FreeNAS. Using this, you can attach any local physical drives to this VM and use software to create a RAID array, which you can then use as a data store. This shouldn’t impact performance too much as long as the FreeNAS system and ESXi are installed on an SSD or equivalent.

-          You could also repurpose an old PC by adding a bunch of hard drives and installing Free NAS or similar.

-          Another option exists that is ideal especially if you are planning on using multiple physical hosts. This option is to buy a NAS then setup either an iSCSI share or an NFS LUN.

-          There are multiple NAS devices available that would work well in a home lab environment and have a small energy footprint. These can range from 2 bay to 8 bay units. If you are using rack mountable equipment, you can easily jump to 12 bays or more.

-          If you go to the mid range models, you will find many NAS devices will offer official VMware compatibility. This usually translates to the ability to create iSCSI shares. Otherwise, almost all NAS units will be able to create NFS shares.

There is obviously a lot to take into account if you are planning on building your own ESXi host/s from scratch. As mentioned earlier, you would need to dedicate a lot of time to making sure the hardware you purchase will suit your needs and allow you to do what you need. I won’t run through individual hardware components that are compatible, but rather I’ll finish on some options for pre-built or brand name computers.

Starting with the outlier, ESXi can be run on Apple hardware. With many people having success with some revisions of the Mac Mini and the Mac Pro. Running ESXi on Apple hardware is a must if you want to virtualize Mac OSX. While there are work-around to get OSX running on non-Apple hardware; Apple’s license agreements state that OSX can only be installed on official Apple hardware.

If virtualizing Mac OSX is not a need for you, there are a large number of options. Starting with small servers like the HP ProLiant MicroServers. These can be configured with Xeon server grade processors, allow up to 16gb of RAM and have 4 SATA bays. It has a built in gigabit NIC and an internal USB port, which can be used to host ESXi. Better yet, these little servers are VMware certified, so they will work out of the box.

Other options could be low-end tower-based servers like some of the Dell PowerEdge versions. These can allow for a bit more flexibility than the smaller form factors like the HP ProLiant MicroServers.

You could also look at Workstation class computers that often come with high-end components and Xeon processors. The drawback with these is that they are often designed for multimedia work and usually come with high-end graphic cards that aren’t needed in an ESXi environment. So they are not always the best value for money.

To summarise, building your own home lab can be a fun process where you can learn a lot. Do some research online, as there is a large community of users who are doing this same thing with almost every combination of hardware available? I hope this information has helped you make an informed decision moving forward and the best of luck in setting up your ideal home lab.