Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

der8auer: 8x Samsung 960 Pro m.2 NVMe in RAID-0 with THREADRIPPER

I think @LinusTech needs this for his gaming machine.

Ketchup is better than mustard.

GUI is better than Command Line Interface.

Dubs are better than subs

Link to post
Share on other sites
50 minutes ago, Trik'Stari said:

I think @LinusTech needs this for his gaming machine.

We'll probably see @LinusTech migrate his all NVMe storage server from the current system using Windows Storage Spaces to AMD EPYC with native NVMe RAID, wouldn't be surprised if he is already talking to AMD about options to do this and how it would stack up namely parity RAIDs.

 

Probably wont read this @LinusTech but did you end up going with Two-Way Mirror or Parity with your configuration? 

 

I've currently got 6 HPE DL380 Gen9 servers with 4 NVMe Write-Intensive SSDs in them each using Two-Way Mirror across the 4 SSDs and performance while great isn't even close to native capability and in some ways worse than just a single one of them outside of Storage Spaces.

 

Edit:

P.S. Not complaining or putting Storage Spaces down this is a common problem for NVMe SSDs for every storage vendor and technology so far. These devices are just so fast that any small addition to latency in the I/O path and additional steps greatly reduces performance.

Link to post
Share on other sites
1 minute ago, leadeater said:

We'll probably see @LinusTech migrate his all NVMe storage server from the current system using Windows Storage Spaces to AMD EPYC with native NVMe RAID, wouldn't be surprised if he is already talking to AMD about options to do this and how it would stack up namely parity RAIDs.

 

Probably wont read this @LinusTech but did you end up going with Two-Way Mirror or Parity with your configuration? 

AMD's setup can only do 1/0/10, so 5 is not available sadly. also Storage Spaces "Raid 5" has terrible write speed and ok read speed.

if you want to annoy me, then join my teamspeak server ts.benja.cc

Link to post
Share on other sites
5 minutes ago, The Benjamins said:

AMD's setup can only do 1/0/10, so 5 is not available sadly. also Storage Spaces "Raid 5" has terrible write speed and ok read speed.

Yea see my edit, also parity on all SSD configurations actually isn't that bad for write. I mean still no where near native or read performance but you can do well over 1GB/s write, IOPs is still the bigger issue though.

Link to post
Share on other sites
1 minute ago, leadeater said:

Yea see my edit, also parity on all SSD configurations actually isn't that bad for write. I mean still no where near native or read performance but you can do well over 1GB/s write.

Ya this is what I got on my HDD set up with some SS tests. It was 4 x 8TB drives (Seagate @ $180)

 

1 HDD

Sequential Read (Q= 32,T= 1) :   215.928 MB/s
Sequential Write (Q= 32,T= 1) :   112.177 MB/s

"Raid 0"

Sequential Read (Q= 32,T= 1) :   650.568 MB/s

Sequential Write (Q= 32,T= 1) :   461.208 MB/

"Raid 1;10"

Sequential Read (Q= 32,T= 1) :   395.798 MB/s
Sequential Write (Q= 32,T= 1) :   250.064 MB/s

"Raid 5"

Sequential Read (Q= 32,T= 1) :   631.123 MB/s
Sequential Write (Q= 32,T= 1) :    62.920 MB/s

if you want to annoy me, then join my teamspeak server ts.benja.cc

Link to post
Share on other sites
16 minutes ago, The Benjamins said:

also Storage Spaces "Raid 5" has terrible write speed and ok read speed.

I'll add some extra context too, I was having a PM conversation with Linus about his NVMe setup and recommended he used Storage Spaces and Two-Way Mirror. His big concern was the loss in usable capacity increasing the $/GB, understandable since you are losing half of your storage.

 

I actually thought he wasn't going to use Storage Spaces at all, only found out when he mentioned it in one of his videos :P. Still think he is looking at alternative options, the performance complaints from his editors likely gets annoying. Sure filling up the SSDs reduces the performance but with the number he has it shouldn't be noticeable to end users of the server.

Link to post
Share on other sites
5 minutes ago, The Benjamins said:

Ya this is what I got on my HDD set up with some SS tests. It was 4 x 8TB drives (Seagate @ $180)

 

1 HDD

Sequential Read (Q= 32,T= 1) :   215.928 MB/s
Sequential Write (Q= 32,T= 1) :   112.177 MB/s

"Raid 0"

Sequential Read (Q= 32,T= 1) :   650.568 MB/s

Sequential Write (Q= 32,T= 1) :   461.208 MB/

"Raid 1;10"

Sequential Read (Q= 32,T= 1) :   395.798 MB/s
Sequential Write (Q= 32,T= 1) :   250.064 MB/s

"Raid 5"

Sequential Read (Q= 32,T= 1) :   631.123 MB/s
Sequential Write (Q= 32,T= 1) :    62.920 MB/s

For what ever reason HDD parity Storage Spaces is particularly garbage, using Tiering or Journal is basically a requirement.

Link to post
Share on other sites
5 minutes ago, leadeater said:

For what ever reason HDD parity Storage Spaces is particularly garbage, using Tiering or Journal is basically a requirement.

Ya Tiering is really cool, but I didn't have extra money for SSD's at the time. I do have 4 2.5" hot swap bays in the front so I can add SSD's when ever.

 

But the Mirror set up I use it working fine for me, but sucks I only have 16TB out of the 32TB

 

EDIT: Can I turn on Tier after the fact and add in a few SSD's with out losing Data? @leadeater

if you want to annoy me, then join my teamspeak server ts.benja.cc

Link to post
Share on other sites
3 minutes ago, The Benjamins said:

Ya Tiering is really cool, but I didn't have extra money for SSD's at the time. I do have 4 2.5" hot swap bays in the front so I can add SSD's when ever.

 

But the Mirror set up I use it working fine for me, but sucks I only have 16TB out of the 32TB

Are you using direct install Windows or HBA passthrough to a VM? I've had issues for a while now that I haven't got around to nailing down but on one of my older servers using HBA passthrough the storage performance sucks however on my newer server with the exact same Storage Spaces configuration the performance is vastly better. Been annoying me for ages.

Link to post
Share on other sites
1 minute ago, leadeater said:

Are you using direct install Windows or HBA passthrough to a VM? I've had issues for a while now that I haven't got around to nailing down but on one of my older servers using HBA passthrough the storage performance sucks however on my newer server with the exact same Storage Spaces configuration the performance is vastly better. Been annoying me for ages.

It is direct install of Windows Server 2012 R2 Datacenter. I was thinking of doing a in place update to 2016 but im scared.

if you want to annoy me, then join my teamspeak server ts.benja.cc

Link to post
Share on other sites
6 minutes ago, The Benjamins said:

It is direct install of Windows Server 2012 R2 Datacenter. I was thinking of doing a in place update to 2016 but im scared.

Upgrading to Server 2016 is worth it just for Multi-Resilient Volumes. You'll be able to tier your HDDs so that it's got both two-way mirror and single/dual parity and all writes always go to two-way mirror tier and data is dynamically moved between the tiers so hot data is always working out of the mirror tier for reads and write.

 

Edit:

Server 2012 R2 moved data between tiers out of band on a schedule based on usage, that's why Server 2016 is so much better.

Link to post
Share on other sites
18 minutes ago, The Benjamins said:

EDIT: Can I turn on Tier after the fact and add in a few SSD's with out losing Data? @leadeater

You can add Journal after the fact but you can't change a virtual disk from non-tiered to tiered, really sucks. Also if your server install doesn't have much 3rd party stuff installed on it you can actually do a clean install and Server 2016 will happily import the existing Storage Spaces pool then you just use powershell to upgrade the pool to Server 2016 feature version. For any VMs etc same deal just re-import.

 

Clean install is much nicer than upgrading, I try to never upgrade if I can help it.

Link to post
Share on other sites
1 minute ago, leadeater said:

You can add Journal after the fact but you can't change a virtual disk from non-tiered to tiered, really sucks. Also if your server install doesn't have much 3rd party stuff installed on it you can actually do a clean install and Server 2016 will happily import the existing Storage Spaces pool then you just use powershell to upgrade the pool to Server 2016 feature version. For any VMs etc same deal just re-import.

 

Clean install is much nicer than upgrading, I tried to never upgrade if I can help it.

That sucks about the tiering

 

That is nice doing a clean install is easy but I am lazy so I may just do a upgrade when I can.

if you want to annoy me, then join my teamspeak server ts.benja.cc

Link to post
Share on other sites
  • 3 months later...
2 hours ago, Azzitude said:

Meanwhile my single m.2 cheap drive ......

Capture.PNG.5ed4839c96909e042b13d4289b17a3ea.PNG

 

TOSHIBA-RD400_512GB_1GB-20180116-2355.png.477a2c4f47daba320a026a62583fca44.png

 

Turn your ram caching off ;)

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×