Dell R620 Disk Raid Performance Issues

[Lurick]

I just wanted to toss this topic out there to make sure I'm not overlooking something obvious with my setup and make sure I'm not missing anything before I move onto something else. Right now my setup is as follows:

Dell R620 with dual E5-2690 CPUs

16x16GB of RAM

H710P (currently in IT mode but I've tried this all in regular non-IT mode too)

8x SSDs (For RAID)

1x HDD (bulk storage)

1x SSD (ESXi install some VM storage)

 

Problem:

I've benchmarked, through dd, hdparm, crystaldiskmark, and other tools, the 8x SSDs as standalone disks and I get expected performance for SATA SSDs. The issue comes in when I try to RAID two or more of the 8x SSDs together. I've put them in RAID 10 and RAID 0 and no matter what I do (explained below) I cannot get any additional performance out of the drives, period, when added to a VM.

 

What I've tried:

Create a RAID 0 on the H710p in Dell firmware mode and passed through the entire disk to the VM as a VMFS6

Create a RAID 0 on the H710p in Dell firmware mode and passed through the entire disk to the VM as a raw disk

Create a RAID 10 on the H710p and pass through as above (VMFS6 and raw disk)

Make each disk as a single disk RAID 0 in Dell firmware mode and pass through entire disk in both VMFS6 and raw disks and make a RAID in the VM

Converted the H710p to IT mode so it's in HBA mode and tried passing through each disk as a VMFS6 and raw disk and creating a software raid in the VM.

 

Am I missing something obvious or is this expected? I even tried wiping my ESXi 7 install and moving down to 6.7 in hopes that maybe on 6.7 a vmklinux driver might work better than the native driver when forcing mpt2sas over lsi_msgpt2 native driver but to no avail.

 

 

Tagging @leadeater as I know you work with servers a good bit.

[leadeater]

You are likely fighting against ESXi, RAID and TRIM support/compatibility issues. You should be able to prove that by passing through the disk without putting a VMFS filesystem on it, pass through the raw disk (you can do this) or pass through an entire HBA/RAID device to the VM.

 

SSDs and ESXi have a long and cursed history, my Samsung 840/850 Pros back in the day just straight up did not like being an ESXi VMFS datastore (this is before UNMAP support) and would over time (short time) end up only doing about 30MB/s.

 

Have a read of these and see if your datastores are properly being detected as SSD/Flash type and UNMAP/TRIM is actually working.

https://www.codyhosterman.com/2016/11/whats-new-in-esxi-6-5-storage-part-i-unmap/

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-BC1A172C-E649-4812-B8B2-A9E45AC97051.html

[Lurick]

12 minutes ago, leadeater said:

You are likely fighting against ESXi, RAID and TRIM support/compatibility issues. You should be able to prove that by passing through the disk without putting a VMFS filesystem on it, pass through the raw disk (you can do this) or pass through an entire HBA/RAID device to the VM.

 

SSDs and ESXi have a long and cursed history, my Samsung 840/850 Pros back in the day just straight up did not like being an ESXi VMFS datastore (this is before UNMAP support) and would over time (short time) end up only doing about 30MB/s.

 

Have a read of these and see if your datastores and properly being detected as SSD/Flash type and UNMAP/TRIM is actually working.

https://www.codyhosterman.com/2016/11/whats-new-in-esxi-6-5-storage-part-i-unmap/

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-BC1A172C-E649-4812-B8B2-A9E45AC97051.html

Yah, I tried passing through each disk as a RMD and no change in RAID on the VM side

Will give those links a look here in a few and report back though.

 

Edit:

I also tried eager and lazy zero on the disks too but to no avail.

[leadeater]

If you can try passing through the entire HBA, that's what I do and it works great. Just a bit of a pain as it disables the ability to do quite a few things to the VM while it's powered on, also don't be dumb like I did once and present storage from the VM back to the host then move the OS VMDK on to that datastore. It was only supposed to be temporary but you know how temporary changes go, it stayed on that datastore and I shut the VM down, RIP that VM  and everything on the datastore forever lol.

[Lurick]

2 minutes ago, leadeater said:

If you can try passing through the entire HBA, that's what I do and it works great. Just a bit of a pain as it disables the ability to do quite a few things to the VM while it's powered on, also don't be dumb like I did once and present storage from the VM back to the host then move the OS VMDK on to that datastore. It was only supposed to be temporary but you know how temporary changes go, it stayed on that datastore and I just the VM down, RIP that VM  and everything on the datastore forever lol.

I've thought about that but what I'm not sure on is if that's possible since the one HBA is connected to all disks in the server including the ESXi install, would that cause any issues?

[leadeater]

1 minute ago, Lurick said:

I've thought about that but what I'm not sure on is if that's possible since the one HBA is connected to all disks in the server including the ESXi install, would that cause any issues?

Yes, you would have to either have a second HBA or use a SATA port for the ESXi OS SSD.

[Lurick]

1 minute ago, leadeater said:

Yes, you would have to either have a second HBA or use a SATA port for the ESXi OS SSD.

Yah, that's what I was thinking, crud. The entire backplane is wired to two 8060 or whatever ports so I'd need to do some funky business in that case

[BloodKnight7]

Just out of curiosity, did you disabled the write/read cache for the SSD when you created the RAID 0/RAID10 drives? with SSDs you get better performance without those.

[Lurick]

On 5/10/2021 at 5:47 PM, BloodKnight7 said:

Just out of curiosity, did you disabled the write/read cache for the SSD when you created the RAID 0/RAID10 drives? with SSDs you get better performance without those.

Yah, I enabled write through and no read back/ahead. I also tried force disabling the cache as well for the drives but to no avail.

 

 

On 5/8/2021 at 11:39 AM, leadeater said:

Yes, you would have to either have a second HBA or use a SATA port for the ESXi OS SSD.

So, I figured out the issue. I had, when I started this adventure, added some samba tweaks to try and help performance out and I never removed those from the file I used between tests. Turns out those were NOT helpful at all on the R630 and hindered performance. I went back and saw a slight bump on the R620 without those tweaks but the R630 really proved to be the key here I think and of course removing Bitdefender that I had installed between the time of ordering and receiving the R630 on my main machine >.>.

Now I get 10Gb/s read and write for sequential operations and about 2Gb/s for random read/writes when passing the drives through as RDM disks.

[leadeater]

4 minutes ago, Lurick said:

Now I get 10Gb/s read and write for sequential operations and about 2Gb/s for random read/writes when passing the drives through as RDM disks.

Yea that sounds a lot better, glad it wasn't hardware/compatibility related

[Lurick]

Just now, leadeater said:

Yea that sounds a lot better, glad it wasn't hardware/compatibility related

Yah, I'm still a bit perplexed on the random performance as I would have expected more from these SSDs. I put the 8 drives in RAID 10 and RAID 0 and no performance improvement was gained by going RAID 0 which has me a bit stumped but meh

[BloodKnight7]

6 hours ago, Lurick said:

Yah, I'm still a bit perplexed on the random performance as I would have expected more from these SSDs. I put the 8 drives in RAID 10 and RAID 0 and no performance improvement was gained by going RAID 0 which has me a bit stumped but meh

yeah... its unfortunate that ESXi is limited in this specific regard, you have to depend on that hardware RAID