Superfast SSD RAID

[Icepick]

Based off your vlog of a FAST 12TB HDD, I have another challenge for you.  We've been given a project of building a storage array of 4TB capable of RW around 2.5 to 3.5 GB per second, as cheap as possible.  So if you could almost double the RW speeds of a spinning disk using a optane SSD what would it be capable of when using it with cheaper SSDs with about 540MB RW and even when trying to use it in a RAID 0.

[NelizMastr]

Optane doesn't do much for regular SSDs. If you want the cheapest possible way of getting those speeds, all you need is 2 2TB 970 Evo's in Raid0. You'll far surpass said speeds even  

 

Or... use 8 500GB 860 Evo's.

[Mikensan]

Since they make 1TB SSDs, and the larger they are the faster their theoretical speeds are. However different workloads will require different tiers of SSD storage. If this is just going to be a file server (SMB) then consumer grade SSDs will work great. If you're going to be running databases, writing large data blocks, 4k r/w/ hypervisor datastores - or any other "enterprise" workloads... "cheap" isn't in the vocabulary.

 

A 4TB enterprise solution on a single shelf/array should cost... $10k-ish? 

6x (Raid 10, just over 4TB)  https://www.newegg.com/Product/Product.aspx?Item=9SIA2F874H6389 alone is $6,000...

[Icepick]

On 6/19/2018 at 9:41 PM, Mikensan said:

Since they make 1TB SSDs, and the larger they are the faster their theoretical speeds are. However different workloads will require different tiers of SSD storage. If this is just going to be a file server (SMB) then consumer grade SSDs will work great. If you're going to be running databases, writing large data blocks, 4k r/w/ hypervisor datastores - or any other "enterprise" workloads... "cheap" isn't in the vocabulary.

 

A 4TB enterprise solution on a single shelf/array should cost... $10k-ish? 

6x (Raid 10, just over 4TB)  https://www.newegg.com/Product/Product.aspx?Item=9SIA2F874H6389 alone is $6,000...

Well granted the information I had at the time your opinion does make perfect sense.  We are talking about a database with millions of transactions running, so we need to speed up those transactions as much as we possibly can to fit more transactions and being able to sell more results.  Without being able to divulge too much information the batches that run on these servers and DBs are what brings in the money.  However this project we have now is just a trail and "scaled" version to see how it works and the budget approved for this test is around ZAR50K (about $4.5K).

[Icepick]

On 6/19/2018 at 3:12 PM, NelizMastr said:

Optane doesn't do much for regular SSDs. If you want the cheapest possible way of getting those speeds, all you need is 2 2TB 970 Evo's in Raid0. You'll far surpass said speeds even  

 

Or... use 8 500GB 860 Evo's.

Agreed, but doing RAID on M.2 drives isn't easy (tongue in cheek), at least the hardware isn't as readily available in ZA.  I'm using M.2s as they are the fastest I could find for the cheapest and a similar PCIe ssd with the same size and speeds is about two to three times the price per GB

[Erkel]

I think you miss understand how the DB world works.

 

If you are hitting disk it is broken.

 

You should be reading out of RAM and writing to NVRAM on the disk subsystem.

What is your read to write ratio.   

basically just size your ram so the database (the active bits) fit into RAM.

 

 

[Mikensan]

5 hours ago, Icepick said:

Well granted the information I had at the time your opinion does make perfect sense.  We are talking about a database with millions of transactions running, so we need to speed up those transactions as much as we possibly can to fit more transactions and being able to sell more results.  Without being able to divulge too much information the batches that run on these servers and DBs are what brings in the money.  However this project we have now is just a trail and "scaled" version to see how it works and the budget approved for this test is around ZAR50K (about $4.5K).

One way to work this is to scale the SSD solution to how much "active" data you would ever have. Even with millions of transaction I suspect at any given hour of the day, the company isn't working on 4TB sized databases. I imagine your budget would be a pinch bigger if that were the case lol.

 

One possible cost effective solution... You could create a very large L2ARC (~1TB) backed by consumer disks (or RI SSDs) and a ZIL (~500gb) backed by enterprise grade or SLC SSDs for write intensive moments. The main 4TB can be backed by consumer SSDs. You can probably just squeeze in under $4.5k USD (assuming you're paying US prices over in ZA).

 

From experience I can tell you sync writes to a consumer SSD is painful.

 

Ultimately the best advice is going to come from a vendor - who can offer support. If the databases are that important it's better to buy premade enterprise gear that comes with support.

[Electronics Wizardy]

5 hours ago, Icepick said:

Well granted the information I had at the time your opinion does make perfect sense.  We are talking about a database with millions of transactions running, so we need to speed up those transactions as much as we possibly can to fit more transactions and being able to sell more results.  Without being able to divulge too much information the batches that run on these servers and DBs are what brings in the money.  However this project we have now is just a trail and "scaled" version to see how it works and the budget approved for this test is around ZAR50K (about $4.5K).

what server are you using? Normally if your using something like a r740xd just put a few nvme ssds in the front and your at the 1 million iops range.

 

What os?

[Icepick]

19 hours ago, Erkel said:

I think you miss understand how the DB world works.

 

If you are hitting disk it is broken.

 

You should be reading out of RAM and writing to NVRAM on the disk subsystem.

What is your read to write ratio.   

basically just size your ram so the database (the active bits) fit into RAM.

 

 

That is not entirely possible as the DB runs at a couple of TBs

[Erkel]

2 hours ago, Icepick said:

That is not entirely possible as the DB runs at a couple of TBs

Run an LSI raid card with an array of SSD's in a 0.5TB Cachecade array.  That will give you 0.5TB of SSD level caching.

You can do that pretty cheaply, as you can get the controllers and cachecade hardware keys cheap secondhand.

 

 

[Icepick]

On 6/22/2018 at 4:57 PM, Electronics Wizardy said:

what server are you using? Normally if your using something like a r740xd just put a few nvme ssds in the front and your at the 1 million iops range.

 

What os?

We haven't decided on a server yet and we'll just build one mean workstation for the trail and then repurpose it when we do get the servers, as for the OS we're running Ubuntu

[Erkel]

I think you would be best to do testing in AWS  or Alibaba first.  Amazon do upto ~4TB of RAM machines,  Start out small and scale up until you hit diminishing returns.   I think you will find that you will need less RAM than you think.  What database are you running?

 

Do not waste you time with a workstation,  know what works first before you throw money at it.

[Icepick]

18 hours ago, Erkel said:

I think you would be best to do testing in AWS  or Alibaba first.  Amazon do upto ~4TB of RAM machines,  Start out small and scale up until you hit diminishing returns.   I think you will find that you will need less RAM than you think.  What database are you running?

 

Do not waste you time with a workstation,  know what works first before you throw money at it.

There are certain constraints we have that prevent us from using and testing on such services else we'd just move over to cloud computing and be done with it and hence my reason for starting this post on the forum to get as much information, theoretical at least, as I can before we go and acquire the hardware only to discover it's money wasted.  As for the DBs that is information I cannot divulge other than we're running it on MySQL.  Should we continue with this trail project we also keeping in mind that we can repurpose the PC for other means within our systems so that it's also still being used and not wasted.

[leadeater]

On 6/22/2018 at 9:14 PM, Icepick said:

Well granted the information I had at the time your opinion does make perfect sense.  We are talking about a database with millions of transactions running, so we need to speed up those transactions as much as we possibly can to fit more transactions and being able to sell more results.  Without being able to divulge too much information the batches that run on these servers and DBs are what brings in the money.  However this project we have now is just a trail and "scaled" version to see how it works and the budget approved for this test is around ZAR50K (about $4.5K).

What type of queries are being run? Are they mostly selects and reading out information with some basic transformations or are there a lot of inserts/updates/writes involved?

 

On 6/23/2018 at 9:13 PM, Icepick said:

That is not entirely possible as the DB runs at a couple of TBs

It doesn't actually matter how large the database is or databases it's the active data set that matters more. Active data stays in server memory so unless your doing entire scans over the full database every time then 99% of the time you are hitting ram and never disk.

 

You also need to look at your data access patterns and query structure and create the appropriate indexes as this will give a far larger performance increase than throwing NVMe like disk performance at it. Brute force performance very rarely works and is the option of last resort after all other reasonable efforts have been done.

 

On 6/20/2018 at 12:41 AM, Icepick said:

We've been given a project of building a storage array of 4TB capable of RW around 2.5 to 3.5 GB per second, as cheap as possible. 

This is actually the wrong metric to look at. You want to be looking at IOPs and you need to know the I/O block size of the application you are using and build a storage solution that best fits that. I believe MySQL uses 16KB block sizes but I'm not a big expert in that DB engine, either way for all DB engines the block sizes are not that large so you won't be getting disk throughput of multiple GB/s outside of running a backup or restore.

 

For reference here is an array of 6 Samsung 850 Pros (RAID 10) and you're interested in the 8.0 to 64.0 values. This is purely indicative and ATTO is a terrible tool to use to benchmark server workload performance but it's quick, IOmeter is much more appropriate but you run those tests over hours not a minute or so.