Teaching an old dog some new tricks: rebuilding my dual Xeon server with NVMe

[maxtch]

I have been with this dual Xeon server for over four years already. It was my VM host server and home gateway for more about two years, than it lied dormant for quite a while until this reactivation. During its dormant years I still upgraded some of its components: back when the processors are cheap I bought two Xeon E5-2680 replacing the origial E5-2620v2, going from 12x Ivy Bridge 2.1GHz cores to 16x Sandy Bridge 2.7GHz cores, a significant upgrade. I replaced the out of support Broadcom 3ware 9750-8i SAS RAID card with a Broadcom MegaRAID 9271-8iCC for the SSD cache capability, added Wi-Fi, Bluetooth and USB 3.0 as PCIe cards and moved it into a rackmount chassis along with my other computers.

 

This story is about adding the NVMe boot drive it currently has. Since this Ivy Bridge platform is way too old, no native NVMe support ever existed, and I have to be creative.

 

First off, current specs (with last update year of this spec in brackets, showing my trickle upgrade strategy of upgrading compters)

• CPU: 2x Intel Xeon E5-2680, 16C/32T total Sandy Bridge @ 2.7/3.5GHz (last updated in 2015)
• RAM: 8x 16GB Kingston KVR DDR3-1600 Registered ECC = 128GB total (original 2013)
• Motherboard: Asus Z9PE-D16C/2L, Intel C602 chipset (original 2013)
• Graphics: Dataland R9 380 4GB (bought in 2015, installed in 2016)
• Wireless: Qualcomm Atheros AR9460 Wi-Fi + Bluetooth 4.0 mini-PCIe module on adapter (installed in 2017)
• USB: Unbranded Chinese USB 3.0 PCIe card (added 2017)
• RAID: Broadcom MegaRAID 9271-8iCC (added 2017)
• HDD: 5x 3TB drives, mixed brands, in RAID 6 (very mixed dates of purchase - earliest was 2012)
• SSD: Kingston SSDNow V300 SATA 240GB, SSD cache drive for the RAID (bought in 2016, bouncing in and out of machines since)
• SSD: Samsung 960 Pro 512GB NVMe on adapter (just added brand new in 2018)

 

For my purposes, this NVMe SSD must be the boot device. My answer: either a BIOS mod, or a boot trampoline using Clover. BIOS mod is the straightforward method that gives me effectively native NVMe support, Clover boot trampoline, while requires a Flash drive, do not have the problem of having the drive sticking out the back of the machine, since this motherboard comes with an onboard USB Type-A connector. So here is my order of attempts: BIOS mod using drivers extracted from a subsequent Asus server board first, BIOS mod using a generic driver next, boot trampoline last. In order not to generate false positives and make test boots faster, I lifted the RAID card out of its slot during the test.

 

In order to flash a modified BIOS into an Asus motherboard, the usual method involves either exploiting a code signature TOCTTOU bug in Asus EZ Flash, or make use of the lack of signature verification in BIOS Flashback. Being a server motherboard, neither trick would likely work. At least for my specific motherboard, there is a way out for this: the BIOS Flash chip, a W25Q64FV, happen to be in a DIP-8 package and is socketed. I have a TL866II Plus universal programmer (and a few Arduino boards if I really want to go the long way,) so as long as I can get a flat binary out of the BIOS editing tool, I can just pull the chip out to flash it externally. Most motherboards comes with srface mounted BIOS chips. so external flashing will require an additional piece of hardware: a programming clip.

 

For the first attempt I followed a Chinese BIOS modding guide, which injected three drivers from Z97 Maximus BIOS into a Z77 board, allowing NVMe boot. That guide required me to extract three drivers: Nvme, NvmeSmm and NvmeInt13 from Asus Z11PA-D8 BIOS. Maybe it is how I inserted the drivers, this trick did not work, the modified BIOS did not pick up the NVMe drive. For the second attempt I tried following this guide. It provided a single, generic NvmExpressDxe driver to be injected. Lo and behold, the modded BIOS detected... a PATA drive? I know for sure that wile I have added various features to this motherboard using PCIe, I never added PATA. This is very likely the NVMe drive, so I popped in my Windows 10 install USB, and voila the install and subsequent boot goes on without a hitch.

 

Since my BIOS mod worked, I never had to deploy the Clover trampoline. Still I feel that I might need to explain the Clover trampoline a bit. Clover is a UEFI bootloader commonly used in loading macOS on non-Apple hardware. However in order to make this work this bootloader have to be capable of loading additional EFI drivers off a disk. (Apple has way too commonly used 16MB, in cases even 32MB Flash chips for their EFI, so they can have the additional space for their various extra drivers not commonly found in regular PCs, which usually only has 8MB Flash for UEFI firmware.) The Clover boot trampoline uses a stripped down copy of Clover to load the NvmExpressDxe driver from a small (256MB) USB stick, and trampoline into the Windows installation. To the BIOS it is booting the USB stick, although it just simply chain-loads Windows.

 

Now, benchmarks. I have some installations going on the background though:

 

Here is my background apps situation:

Yes that is Eclipse and Arduino IDE, so chant with me:

Quote

Developers, developers, developers, develoeprs;

Developers, developers, developers, develoeprs...

- Steve Ballmer, 2000

 

[maxtch]

I wonder if it is a good idea to have the NVMe adapter card fixed on the side panel of my motherboard using double-sided tape, and using an PCIe extender cable to reach one of two slots blocked by the graphics card but avoiding hitting it. This way I can free up the PCIe slot the card currently occupies so I can put my capture card back.