der8auer: 8x Samsung 960 Pro m.2 NVMe in RAID-0 with THREADRIPPER

[leadeater]

50 minutes ago, Trik'Stari said:

I think @LinusTech needs this for his gaming machine.

We'll probably see @LinusTech migrate his all NVMe storage server from the current system using Windows Storage Spaces to AMD EPYC with native NVMe RAID, wouldn't be surprised if he is already talking to AMD about options to do this and how it would stack up namely parity RAIDs.

 

Probably wont read this @LinusTech but did you end up going with Two-Way Mirror or Parity with your configuration? 

 

I've currently got 6 HPE DL380 Gen9 servers with 4 NVMe Write-Intensive SSDs in them each using Two-Way Mirror across the 4 SSDs and performance while great isn't even close to native capability and in some ways worse than just a single one of them outside of Storage Spaces.

 

Edit:

P.S. Not complaining or putting Storage Spaces down this is a common problem for NVMe SSDs for every storage vendor and technology so far. These devices are just so fast that any small addition to latency in the I/O path and additional steps greatly reduces performance.

[The Benjamins]

1 minute ago, leadeater said:

We'll probably see @LinusTech migrate his all NVMe storage server from the current system using Windows Storage Spaces to AMD EPYC with native NVMe RAID, wouldn't be surprised if he is already talking to AMD about options to do this and how it would stack up namely parity RAIDs.

 

Probably wont read this @LinusTech but did you end up going with Two-Way Mirror or Parity with your configuration? 

AMD's setup can only do 1/0/10, so 5 is not available sadly. also Storage Spaces "Raid 5" has terrible write speed and ok read speed.

[leadeater]

5 minutes ago, The Benjamins said:

AMD's setup can only do 1/0/10, so 5 is not available sadly. also Storage Spaces "Raid 5" has terrible write speed and ok read speed.

Yea see my edit, also parity on all SSD configurations actually isn't that bad for write. I mean still no where near native or read performance but you can do well over 1GB/s write, IOPs is still the bigger issue though.

[The Benjamins]

1 minute ago, leadeater said:

Yea see my edit, also parity on all SSD configurations actually isn't that bad for write. I mean still no where near native or read performance but you can do well over 1GB/s write.

Ya this is what I got on my HDD set up with some SS tests. It was 4 x 8TB drives (Seagate @ $180)

 

1 HDD

Sequential Read (Q= 32,T= 1) :   215.928 MB/s
Sequential Write (Q= 32,T= 1) :   112.177 MB/s

"Raid 0"

Sequential Read (Q= 32,T= 1) :   650.568 MB/s

Sequential Write (Q= 32,T= 1) :   461.208 MB/

"Raid 1;10"

Sequential Read (Q= 32,T= 1) :   395.798 MB/s
Sequential Write (Q= 32,T= 1) :   250.064 MB/s

"Raid 5"

Sequential Read (Q= 32,T= 1) :   631.123 MB/s
Sequential Write (Q= 32,T= 1) :    62.920 MB/s

[leadeater]

16 minutes ago, The Benjamins said:

also Storage Spaces "Raid 5" has terrible write speed and ok read speed.

I'll add some extra context too, I was having a PM conversation with Linus about his NVMe setup and recommended he used Storage Spaces and Two-Way Mirror. His big concern was the loss in usable capacity increasing the $/GB, understandable since you are losing half of your storage.

 

I actually thought he wasn't going to use Storage Spaces at all, only found out when he mentioned it in one of his videos . Still think he is looking at alternative options, the performance complaints from his editors likely gets annoying. Sure filling up the SSDs reduces the performance but with the number he has it shouldn't be noticeable to end users of the server.

[leadeater]

5 minutes ago, The Benjamins said:

Ya this is what I got on my HDD set up with some SS tests. It was 4 x 8TB drives (Seagate @ $180)

 

1 HDD

Sequential Read (Q= 32,T= 1) :   215.928 MB/s
Sequential Write (Q= 32,T= 1) :   112.177 MB/s

"Raid 0"

Sequential Read (Q= 32,T= 1) :   650.568 MB/s

Sequential Write (Q= 32,T= 1) :   461.208 MB/

"Raid 1;10"

Sequential Read (Q= 32,T= 1) :   395.798 MB/s
Sequential Write (Q= 32,T= 1) :   250.064 MB/s

"Raid 5"

Sequential Read (Q= 32,T= 1) :   631.123 MB/s
Sequential Write (Q= 32,T= 1) :    62.920 MB/s

For what ever reason HDD parity Storage Spaces is particularly garbage, using Tiering or Journal is basically a requirement.

[The Benjamins]

5 minutes ago, leadeater said:

For what ever reason HDD parity Storage Spaces is particularly garbage, using Tiering or Journal is basically a requirement.

Ya Tiering is really cool, but I didn't have extra money for SSD's at the time. I do have 4 2.5" hot swap bays in the front so I can add SSD's when ever.

 

But the Mirror set up I use it working fine for me, but sucks I only have 16TB out of the 32TB

 

EDIT: Can I turn on Tier after the fact and add in a few SSD's with out losing Data? @leadeater

[leadeater]

3 minutes ago, The Benjamins said:

Ya Tiering is really cool, but I didn't have extra money for SSD's at the time. I do have 4 2.5" hot swap bays in the front so I can add SSD's when ever.

 

But the Mirror set up I use it working fine for me, but sucks I only have 16TB out of the 32TB

Are you using direct install Windows or HBA passthrough to a VM? I've had issues for a while now that I haven't got around to nailing down but on one of my older servers using HBA passthrough the storage performance sucks however on my newer server with the exact same Storage Spaces configuration the performance is vastly better. Been annoying me for ages.

[The Benjamins]

1 minute ago, leadeater said:

Are you using direct install Windows or HBA passthrough to a VM? I've had issues for a while now that I haven't got around to nailing down but on one of my older servers using HBA passthrough the storage performance sucks however on my newer server with the exact same Storage Spaces configuration the performance is vastly better. Been annoying me for ages.

It is direct install of Windows Server 2012 R2 Datacenter. I was thinking of doing a in place update to 2016 but im scared.

[leadeater]

6 minutes ago, The Benjamins said:

It is direct install of Windows Server 2012 R2 Datacenter. I was thinking of doing a in place update to 2016 but im scared.

Upgrading to Server 2016 is worth it just for Multi-Resilient Volumes. You'll be able to tier your HDDs so that it's got both two-way mirror and single/dual parity and all writes always go to two-way mirror tier and data is dynamically moved between the tiers so hot data is always working out of the mirror tier for reads and write.

 

Edit:

Server 2012 R2 moved data between tiers out of band on a schedule based on usage, that's why Server 2016 is so much better.

[leadeater]

18 minutes ago, The Benjamins said:

EDIT: Can I turn on Tier after the fact and add in a few SSD's with out losing Data? @leadeater

You can add Journal after the fact but you can't change a virtual disk from non-tiered to tiered, really sucks. Also if your server install doesn't have much 3rd party stuff installed on it you can actually do a clean install and Server 2016 will happily import the existing Storage Spaces pool then you just use powershell to upgrade the pool to Server 2016 feature version. For any VMs etc same deal just re-import.

 

Clean install is much nicer than upgrading, I try to never upgrade if I can help it.

[The Benjamins]

1 minute ago, leadeater said:

You can add Journal after the fact but you can't change a virtual disk from non-tiered to tiered, really sucks. Also if your server install doesn't have much 3rd party stuff installed on it you can actually do a clean install and Server 2016 will happily import the existing Storage Spaces pool then you just use powershell to upgrade the pool to Server 2016 feature version. For any VMs etc same deal just re-import.

 

Clean install is much nicer than upgrading, I tried to never upgrade if I can help it.

That sucks about the tiering

 

That is nice doing a clean install is easy but I am lazy so I may just do a upgrade when I can.

[marcintosch]

The video is back online, I edited the original post (:

[Azzitude]

Meanwhile my single m.2 cheap drive ......

 

 

[leadeater]

2 hours ago, Azzitude said:

Meanwhile my single m.2 cheap drive ......

 

 

Turn your ram caching off