brwainer's Server2016 and Veeam Testing and Log

[brwainer]

With Windows Server 2016 release right around the corner, I'm going to be overhauling my setup.

 

Goal:

• Hypervisor: HyperV 2016 (guests will all stay at 2012R2/CentOS7 for now)
• Storage: Storage Spaces
• Boot: USB thumb drive, or maybe SataDOM
• Replication and backup via Veeam Backup Essentials

Current Hardware:

Server A "Reptar"

• Location: My house (southeastern VA)
• Chassis: Norco RPC2304
• Mobo: Supermicro X10SLM-F
• CPU: Xeon E3-1230v3
• RAM: 32GB (4x8GB) DDR3-1600 ECC
• Boot: 850 Evo 256GB
• SSDs: 2x 850 Evo 256GB, 2x MX100 512GB
• HDDs: 2x HGST NAS 4TB, 2x HGST NAS 3TB

Server B "Disney"

• Location: PA, west of Philladelphia
• Chassis: Silverstone DS380
• Mobo: Asrock E2C224D4i-14S
• CPU: Xeon E3-1220v3
• RAM: 32GB (4x8GB) DDR3-1600 ECC
• Boot: Trion 100 120GB
• SSDs: 2x Trion 100 240GB, 2x Force LE 480GB
• HDDs: 2x WD Red 3TB, 1x Seagate Barracuda 3TB, 1x WD Red 2TB, 2x WD Red 2.5" 1TB
• GPU: GTX 960 2GB (my uncle plays games directly on the server, mainly Skyrim. He does not browse the internet on the server except the Nexus mod website)

Immediate future (parts already ordered except the chassis, ordering that on next payday)

Server C "Mountain"

• Location: My house/migratory
• Chassis: Supermicro 813MTQ-350CB
• Mobo: Asrock E3C224D2i (this used to be in Disney)
• CPU: Xeon E3-1220v3
• RAM: 16GB (2x8GB) DDR3-1600 ECC
• Boot: 16GB USB, or 32GB Sata SSD (not SataDOM, just lying around)
• SSDs: none for now, hoping to reclaim the boot SSDs from the other servers
• HDDs: 2x HGST NAS 5TB

General upgrade/rollout strategy

1. Build Mountain
2. Use Mountain to test HyperV boot from USB, or fallback to 32GB SSD (I've set up a thumb drive with HyperV 2012R2 and 2016TP5 per instructions online, but it neither will boot on my desktop - I get a "Inaccesible_boot_device" BSOD after Windows tries to boot for ~45 seconds. Research indicates this is because the OS doesn't have USB drivers for my mobo. I will be retrying this once the server is built)
3. Migrate VMs from Reptar to Mountain. Rebuild Reptar with HyperV 2016, attempting to change boot medium to USB if I can. Migrate VMs back.
4. Move Mountain to PA (I wish I had two systems named Mountain so I could be "moving mountains"). Repeat step 3 with Disney. As part of this process, I need to move my uncle's game and login from the Host to a guest VM.
5. Move Mountain back to VA. Set up Veeam Backup and Replication. Reptar will replicate and back up its VMs to Mountain, and VMs will also replicate to Disney. Disney will replicate its VMs to Mountain, and back up to itself. Probably need to replace the 1TB 2.5" Reds with larger drives.
6. Done?

[brwainer]

Reserved. Also tagging @leadeater

[WhatsCS]

How much have you used 2016? From the few months of using it I found a few issues with ICS and Hyper-V but I haven't used it recently, only when TP5 first was released. How many VM's are you running, I'm curious about the performance of the E3 line. I've only worked with an E5-2440 v2 (I think) and it's... okay, I think I have a disk bottleneck though so I can't really confirm whether or not it's a good cpu for virtualization.

[brwainer]

11 minutes ago, WhatsCS said:

How much have you used 2016? From the few months of using it I found a few issues with ICS and Hyper-V but I haven't used it recently, only when TP5 first was released. How many VM's are you running, I'm curious about the performance of the E3 line. I've only worked with an E5-2440 v2 (I think) and it's... okay, I think I have a disk bottleneck though so I can't really confirm whether or not it's a good cpu for virtualization.

I used TP4 in a VM for a few hours, and same for TP5. I'm confident that any issues you ran into will be fixed upon release, or the within the first month. I'm excited that nearly even limitation with 2012R2 that has bugged me is going away, primarily around Storage Spaces and HyperV.

 

Reptar has 9 Windows 2012 R2 VMs, and 2 CentOS 7 VMs. Disney has 5 Windows VMs and 1 CentOS VM. I've been happy with the performance of the CPUs, Handbrake in a Windows VM on Reptar can process a Bluray rip at about 80% of the speed of my i5-4690k desktop. The CPU in Disney is a little underpowered to be used for gaming, I hope someday to pick up a used E3-1231v3 or higher to drop in there - once they start getting sold off.

[WhatsCS]

8 minutes ago, brwainer said:

I'm confident that any issues you ran into will be fixed upon release, or the within the first month.

They probably will be, as the issue with my install seemed to be a very unique issue when compared to the 30 or so installs we had. It was mostly just ICS's DHCP server messing up, either the gateway assignment messed up some how or the actual assigning of IP's didn't work. Simple fix was changing the IP Block it was using (iirc it was from 192.168.137.xxx to 138.xxx).

I'll have to start looking into the E3 line if you're having that much success with VM's though I guess it would make sense, iirc E3 is just i7 locked and changed up a little bit correct? 

[brwainer]

1 hour ago, WhatsCS said:

They probably will be, as the issue with my install seemed to be a very unique issue when compared to the 30 or so installs we had. It was mostly just ICS's DHCP server messing up, either the gateway assignment messed up some how or the actual assigning of IP's didn't work. Simple fix was changing the IP Block it was using (iirc it was from 192.168.137.xxx to 138.xxx).

I'll have to start looking into the E3 line if you're having that much success with VM's though I guess it would make sense, iirc E3 is just i7 locked and changed up a little bit correct? 

E3 can either be i5 based (e.g. E3-1220v3 which has quad core, no HT) or i7 based (E3-1231v3 which has quad core with HT). It makes sense if you look at all the CPUs that support ECC - at the low end they have i3s and some Pentiums, then E3 for i5/i7, and E5 for 2011v3. I don't think E7 has a consumer counterpart

[leadeater]

You actually paid for Veeam for home use? How much that set you back? Also not something I have done any testing of yet but Windows Server 2016 (Datacenter) has block level storage replication, Storage Replica. May or may not be useful to you.

[brwainer]

45 minutes ago, leadeater said:

You actually paid for Veeam for home use? How much that set you back? Also not something I have done any testing of yet but Windows Server 2016 (Datacenter) has block level storage replication, Storage Replica. May or may not be useful to you.

Just using the free version of Veeam One for HyperV monitoring now.... but a 2 CPU socket (counting only sockets that VMs are running on) license of Veeam Backup Essentials (Veeam One plus Backup and Replica, packaged for small businesses) costs $950 one-time (including one year of support and updates) or $568 per year. The price is a bit high for essentially a compicated home lab, but considering I'm still using the Windows licenses I got for free during college, my software investment cost is still basically zero. I'm thinking about starting to do some freelance work, have a possible first client, and if that did come about I'd try to get the Microsoft Action Pack.

 

I haven't looked much into Storage Replica because it solves a problem I don't have. The Microsoft VM replication and Backup have been a bit too limited and easy to create issues with... I have 2 VMs that refuse to back up at all on one machine, but their replicas and all the other VMs of the same config back up fine. I've basically redone everything except amke the VMs from scratch and rebuilding the host OS. That's just one of the problems I've run into with the Microsoft tools. I'm sure Veeam has it's issues, but I haven't heard of anything major - and I sure would love to be able to open a ticket for issues

[leadeater]

Microsoft Action Pack sounds rather nice, first I've heard of it. Also pity I don't have access to my Veeam resale account anymore, could have gotten it for you cheaper (maybe).

 

Veeam is awesome for VM backups, they have nailed it perfectly.

[brwainer]

I'm thinking a bit ahead of myself, but I'm considering topologies for the Veeam Backup and Replication.

 

Current situation:

• Both VM Hosts replicate all their VMs to the other (HyperV Replication)
• Both hosts run complete backups to local storage (Windows Server Backup)

The new setup will have storage on Mountain and Disney as backup repositories. All VMs should end up with backups (run daily, or a few times a day) and replicas (run several times an hour, at least for Key VMs) on both Disney and Mountain. There are two ways I see to set this up:

 

Possible setup #1: Primary network traffic is Replication

• Reptar replicates all VMs to both Disney and Mountain.
• Disney replicates all VMs to Mountain.
• Disney and Mountain do backups of all the VMs
• Pro: All VMs end up with an RPO of equal to or less than the replication interval
• Con: Traffic is constantly being sent. This is more of an issue with the VMs hosted on Disney, because that site has a 50/10 connection (the site with Reptar and Disney is 50/50). With the current setup, traffic from Reptar to Disney is ~5Mbps constant, and traffic from Disney to Reptar is ~3Mbps constant.

Possible setup #2: Primary network traffic is Backups

• Reptar replicates only the key VMs to both Disney and Mountain, and other VMs are backed up to Disney and Mountain.
• Disney replicates only the key VMs to Mountain, and other VMs are backed up to Mountain.
• Disney and Mountain create replicas of the non-key VMs from the backup files
• Pro: much reduced network traffic
• Con: non-key VMs have an RPO equal to the backup interval

Anyone have opinions about which way that they would go in this scenario? I can even go a bit crazier and have 3 or more tiers of VMs, based on what their target RPO should be. So "Tier 1" might be replicated every 5 minutes, "Tier 2" might be replicated or backed up every 6 hours, and "Tier 3" might be backed up daily.

[brwainer]

Today was a good day, with one very saddening failure. First off, the parts for the new server all arrived (except the chassis, which arrives Friday). Right now the system is installed in a spare desktop computer case.

 

First test: Booting HyperV 2016TP5 off of USB: Passed with ease

Second test: Setting up Discrete Device Assignment: Failed - apparently this motherboard's chipset could support SR-IOV and the other required features (specifically having ACS support on the PCIe controllers) but the BIOS lacks it. Checked both my in-production systems (the ones I actually want to do GPU passthrough on) and they lack it as well - even the Supermicro board! I've opened a case with Asrock to see if they can support it, but I'm expecting to hear no.

 

I may have to see what the performance is on RemoteFX in Skyrim to appease my uncle... otherwise it looks like he'll be playing his game directly on the host for the foreseeable future.

[leadeater]

29 minutes ago, brwainer said:

Second test: Setting up Discrete Device Assignment: Failed - apparently this motherboard's chipset could support SR-IOV and the other required features (specifically having ACS support on the PCIe controllers) but the BIOS lacks it. Checked both my in-production systems (the ones I actually want to do GPU passthrough on) and they lack it as well - even the Supermicro board! I've opened a case with Asrock to see if they can support it, but I'm expecting to hear no.

 

I may have to see what the performance is on RemoteFX in Skyrim to appease my uncle... otherwise it looks like he'll be playing his game directly on the host for the foreseeable future.

Are you trying to do device passthrough? SR-IOV is kinda different, I use it on my Intel X540-T1 to create multiple virtual driver level 10Gb nics to assign to VMs rather than using VMXNET3. Device passthrough only required VT-d so unless the kit you got is rather old it should be supported, LGA1366 has it.

 

As for Skyrim I suspect it's not going to work well enough performance wise but the real killer is mouse input, FPS/relative mouse position input straight up is broken.

[brwainer]

1 hour ago, leadeater said:

Are you trying to do device passthrough? SR-IOV is kinda different, I use it on my Intel X540-T1 to create multiple virtual driver level 10Gb nics to assign to VMs rather than using VMXNET3. Device passthrough only required VT-d so unless the kit you got is rather old it should be supported, LGA1366 has it.

 

As for Skyrim I suspect it's not going to work well enough performance wise but the real killer is mouse input, FPS/relative mouse position input straight up is broken.

I'm trying to do Discrete Device Assignment, which is the new feature in 2016 to attach a PCIe device directly to a host. It does require VT-d, which my processor and motherboard support, but they also require ACS, which is some sort of security system for PCIe devices to prevent crosstalk. Apparently ACS is commonly enabled as a feature for supporting SR-IOV.

 

Asrock has responded to my support ticket with a Bios version that supposedly supports SR-IOV - I'll try it when I get a chance.

[leadeater]

3 minutes ago, brwainer said:

I'm trying to do Discrete Device Assignment, which is the new feature in 2016 to attach a PCIe device directly to a host. It does require VT-d, which my processor and motherboard support, but they also require ACS, which is some sort of security system for PCIe devices to prevent crosstalk. Apparently ACS is commonly enabled as a feature for supporting SR-IOV.

 

Asrock has responded to my support ticket with a Bios version that supposedly supports SR-IOV - I'll try it when I get a chance.

A right of course, forgot you were using Hyper-V. Wonder if my servers support device passthrough then.

[brwainer]

Just now, leadeater said:

A right of course, forgot you were using Hyper-V. Wonder if my servers support device passthrough then.

Yeah sticking with Storage Spaces/HyperV for now. Moving my data to another software defined storage system would be easy, but migrating my VMs is not something I would like to do. But since Microsoft is being very safe about ensuring that device passthrough doesn't cause instability or open an attack vector, I may be forced to move to something else if I can't get DDA working.

[Mikensan]

Note sure if this helps, but was googling a way to override ACS checks (like how KVM allows you to) for Server 2016... came across this:

https://blogs.technet.microsoft.com/virtualization/2015/11/20/discrete-device-assignment-machines-and-devices/

 

I feel like there should be away around it since it's a security not functionality feature.

 

thinking maybe instead of "Dismount-VMHostAssignableDevice -force" it could be something like "Mount-VMHostAssignableDevice -force"?

[brwainer]

12 hours ago, Mikensan said:

Note sure if this helps, but was googling a way to override ACS checks (like how KVM allows you to) for Server 2016... came across this:

https://blogs.technet.microsoft.com/virtualization/2015/11/20/discrete-device-assignment-machines-and-devices/

 

I feel like there should be away around it since it's a security not functionality feature.

 

thinking maybe instead of "Dismount-VMHostAssignableDevice -force" it could be something like "Mount-VMHostAssignableDevice -force"?

Definitely have tried the -force, and read that exact page  thanks for trying to help though!

 

 

On the BIOS front - flashed the BIOS Asrock sent me, the option to enable SR-IOV showed up in BIOS, but in Windows, the errors are all the same. I did a cold boot (unplug PSU for 30 seconds) because I heard that some systems required that to enable SR-IOV. I told the Asrock support guy, he responded that he would have to check on the issue with the BIOS team. So I'm waiting for them to get back to me.

[Mikensan]

8 hours ago, brwainer said:

Definitely have tried the -force, and read that exact page  thanks for trying to help though!

 

 

On the BIOS front - flashed the BIOS Asrock sent me, the option to enable SR-IOV showed up in BIOS, but in Windows, the errors are all the same. I did a cold boot (unplug PSU for 30 seconds) because I heard that some systems required that to enable SR-IOV. I told the Asrock support guy, he responded that he would have to check on the issue with the BIOS team. So I'm waiting for them to get back to me.

I have to give props to asrock for going the length for you, that's pretty cool. Was expecting them to either not reply or say "sorry about your luck." I'm curious how this turns out, sorry the -force didn't work, maybe M$ will give an official method of bypassing it in a future/final release.

[brwainer]

I got the chassis and put all the parts in, but I never knew how oddly proprietary Supermicro makes their cases. I was prepared for having to use a breakout cable for the front panel connector, but the riser card is aligned with what would be the second expansion slot, which an mITX board (which this chassis is listed as recommended for) doesn't have. Sure you can fit an mATX board in here, but instead I'm going to go with a flexible PCIe extender.

 

No changes on the software front, still waiting on Asrock. I suspect they hadn't tested this motherboard and BIOS with Server 2016, or at least not SR-IOV. I sent them my steps to recreate the error (install HyperV Server 2016TP5, and run a single command)

 

Edit: Obviously this isn't in my rack yet - A) I don't want to put it in until I have the PCIe extender, and B) I actually need to rearrange the rack a little - It's full and I'm going to get rid of either a cable management row, or a JBOD that I haven't actually installed any drives in yet.

[brwainer]

Lots of updates!

 

With the General Release of Server 2016, I started my migration. Currently, all the VMs are living on the new server, Mountain, but I am running into significant performance issues since Mountain doesn't have an SSD tier. It's so bad that I've turned off some of the less vital VMs, which is something I was trying to avoid.

 

Current problems:

• Mountain doesn't fit in my rack, without some extra parts. The rails that came with the chassis are meant for deeper racks, so I have ordered the appropriate short depth outer rails. Also, the chassis is so deep that with a normal power cable, it can't move back all the way. I've ordered a 90 degree power cable in hopes that it gives enough clearance to insert the chassis fully. Thankfully, the door still closes, and Mountain is sitting happily on top of my SAS JBOD which is securely mounted to the rack with 12 screws.
• One of the ethernet ports on Mountain is only linking at 100Mb, and only does that after a 30 second handshaking with the switch. I'm assuming it's a wire issues, or maybe a connector is loose. I'll look at this when I fix the mounting for Mountain.
• I went to install the new USB boot drive into Reptar and discovered that on this motherboard/chassis combination, the USB port is actually underneath the hard drive cages, meaning that there is about 1/2" of clearance. This means the USB drives I have on hand will not fit. I've ordered two different ones for use, a Samsung Fit and a Sandisk Ultra Fit. Either will work, but I'm actually curious to find out which one performs better.
• Asrock hasn't gotten back to me about the GPU passthrough issue (but I also haven't asked for any updates). After reading up on this some more, I suspect that these motherboards lack the ACS root that Windows Server requires in order to allow Discrete Device Assignment. Therefore I have given up on this path and will be building my uncle an entry level gaming PC, to be installed when I go up there to migrate that server to 2016. We're looking at a selection of chassis options that would fit right above or below the server (basically they are all mini-ITX cube systems, because we only have 12" height available)
• Each of the servers has a bit of an issue storage-wise:
  • Mountain: the 2x 3TB drives are basically full, and the 2x 4TB drives have 1TB free on each of them - this is because almost every virtual disk I've made is in RAID10 and is therefore using equal amounts of space on each drive.
  • Disney: the Hard Drives are basically full, because this server has 2TB less storage than Mountain, but has to store the same amount of data.
  • Reptar: the 2x 5TB drives are great, but the other two drives in the system are old and also are only desktop rated. They also only contribute 820GB combined.
  • Solution: buy 2x 4TB NAS drives. Install them in Mountain in place of the 2x 3TB drives. Install the 2x 3TB drives in Disney, replacing the 2x 1TB drives. Install the 2x 1TB drives in Reptar. Final drive configuration afterwards will be:
    • Mountain: 4x 4TB drives, 16TB raw
    • Disney: 5x 3TB drives and 1x 2TB, 17TB raw
    • Reptar: 2x 5TB drives and 2x 1TB, 12TB raw

 

 

Parts on order and next steps:

• USB flash drives for Reptar
  • I will do performance tests on these and pick the one that has better random read IOPS for installation in Reptar. After Reptar is set up I will migrate the VMs back over. For now I am keeping all the backups as well as the boot drive from the 2012 R2 installation, until I feel confident in the new setup.
• Shorter rails and 90 degree power cable for Mountain
  • Mountain will have to be powered down and unmounted. This will have to wait until the VMs are back running on Reptar.
  • also TS the ethernet link issue
• 2x 4TB drives and parts for new gaming desktop - not ordered yet

[Whaler_99]

Just one thing I will add to this, I found this out this week building a 2016 Server - most backup software has no support yet for 2016, period.

 

We use Altaro Backup (check it out, free to back up to guests) and they are saying no support till at least end of December. I checked the Veeam site as well and nothing listed either in regards to support for 2016 yet. 

 

I installed Altaro, thinking it can't be much different than 2012 R2, would still work, just not official supported. Altaro won't even recognize that Hyper-V is installed on my 2016 box.

[leadeater]

18 minutes ago, Whaler_99 said:

Just one thing I will add to this, I found this out this week building a 2016 Server - most backup software has no support yet for 2016, period.

 

We use Altaro Backup (check it out, free to back up to guests) and they are saying no support till at least end of December. I checked the Veeam site as well and nothing listed either in regards to support for 2016 yet. 

 

I installed Altaro, thinking it can't be much different than 2012 R2, would still work, just not official supported. Altaro won't even recognize that Hyper-V is installed on my 2016 box.

Microsoft has completely changed the entire storage sub system in Server 2016 and made huge changes to the core of Hyper-V, we use at lot of Nutanix (ESXi mind you) and that also doesn't support Server 2016 Hyper-V yet as far as I'm aware.

 

You should be able to take crash consistent VM backups of Server 2016 without file level recovery, in theory. I'll take a backup of a test Server 2016 VM on monday with commvault and see what happens, need to do it anyway.

[brwainer]

1 hour ago, Whaler_99 said:

Just one thing I will add to this, I found this out this week building a 2016 Server - most backup software has no support yet for 2016, period.

 

We use Altaro Backup (check it out, free to back up to guests) and they are saying no support till at least end of December. I checked the Veeam site as well and nothing listed either in regards to support for 2016 yet. 

 

I installed Altaro, thinking it can't be much different than 2012 R2, would still work, just not official supported. Altaro won't even recognize that Hyper-V is installed on my 2016 box.

 

1 hour ago, leadeater said:

Microsoft has completely changed the entire storage sub system in Server 2016 and made huge changes to the core of Hyper-V, we use at lot of Nutanix (ESXi mind you) and that also doesn't support Server 2016 Hyper-V yet as far as I'm aware.

 

You should be able to take crash consistent VM backups of Server 2016 without file level recovery, in theory. I'll take a backup of a test Server 2016 VM on monday with commvault and see what happens, need to do it anyway.

For now I am using my old strategy of HyperV replica between sites, and Windows Server Backup at each site to back up all my VMs. Both my primary server and my backup server at Site A are on hyperV 2016 already, I've made the storage pools be 2016 feature level, but leaving the VMs at 2012R2 level (config version 5.0) for now, otherwise they wouldn't be able to replicate to Site B. Site B will be stuck on 2012R2 until I have both a replacement gaming rig for my uncle, and travel out there to do the upgrade.

 

Once Veeam 9.5 comes out I will buy their small business package and start using it for backup and replication. My wallet is thanking me for having to wait on this move anyway

[Guest]

 

On 30/10/2016 at 2:40 AM, Whaler_99 said:

Just one thing I will add to this, I found this out this week building a 2016 Server - most backup software has no support yet for 2016, period.

Yeah had a few clients of ours with similar issues.

On 30/10/2016 at 4:17 AM, brwainer said:

Once Veeam 9.5 comes out I will buy their small business package and start using it for backup and replication. My wallet is thanking me for having to wait on this move anyway

Depending on what you do for work you could always lodge for NFR pricing through Veeam? They're normally pretty good with that stuff.

[brwainer]

17 minutes ago, Windspeed36 said:

 

Yeah had a few clients of ours with similar issues.

Depending on what you do for work you could always lodge for NFR pricing through Veeam? They're normally pretty good with that stuff.

Wish I could but we're in the cable/ISP business and I'm nowhere near the few servers we virtualize... Although if I was I think I would have been able to prevent the catastrophic array failure that somehow took down nearly all of our internal systems for days. We basically turned into the most lax ISP you have ever seen, since we had no login system to authentice users against. Anyway, I can't honestly say that my homelab would be practice for, or related to, my work.