Viking Technology ships 50TB SSD for the datacenter

[Misanthrope]

I was wondering why we weren't cramming a ton of chips into a 3.5" sized drive. I think this is a fairly significant leap forward since a full 2A rack filled with these means you replace like 10 other servers and a bunch of money in electricity and other hardware upkeep, as crazy expensive as they'll be it still probably should be offset relatively soon if viking can have significant supply.

[biotoxin]

On 7/15/2017 at 3:39 AM, leadeater said:

What makes you say that? Storage Spaces Direct only works on ReFS and Veeam uses it now instead of deduplication for it's awesome metadata capabilities.

they've (ms general) started moving away from it on backend, I guess they're developing yet another file system, something to do with virtualization and speed, apparently v2 didn't make enough improvements

they also haven't had much development of it or progress on fixing any of it's huge problems, e.g. compression & dedupe

 

they've actually made it clear it won't be replacing ntfs like we expected for non-server applications

signs point to they're doing something different

and maybe this is just a goof, but it might indicate something more is that their onedrive service and I think another application, no longer support ReFS drives, recent news though so I'll wait to see where it goes

 

looked into btrfs though for handling this and it might actually work out depending on coalesced writes.

 

[leadeater]

28 minutes ago, biotoxin said:

they've (ms general) started moving away from it on backend, I guess they're developing yet another file system, something to do with virtualization and speed, apparently v2 didn't make enough improvements

they also haven't had much development of it or progress on fixing any of it's huge problems, e.g. compression & dedupe

 

they've actually made it clear it won't be replacing ntfs like we expected for non-server applications

signs point to they're doing something different

and maybe this is just a goof, but it might indicate something more is that their onedrive service and I think another application, no longer support ReFS drives, recent news though so I'll wait to see where it goes

 

looked into btrfs though for handling this and it might actually work out depending on coalesced writes.

 

No dedup is a bit of a slap in the face considering ntfs has it and isn't an unreasonable expectation of a new file system. It's something that really annoys me and lots of people keep asking for that feature.

[landem]

Deduplication on ReFS is contained in the current preview of Windows Server 2016: "Data Deduplication available for ReFS"
https://blogs.windows.com/windowsexperience/2017/07/13/announcing-windows-server-insider-preview-build-16237/

[leadeater]

38 minutes ago, landem said:

Deduplication on ReFS is contained in the current preview of Windows Server 2016: "Data Deduplication available for ReFS"
https://blogs.windows.com/windowsexperience/2017/07/13/announcing-windows-server-insider-preview-build-16237/

Awesome thanks for that, another not worthy feature is "Space efficiency with ReFS Compaction".

[NLD1st]

On 11/07/2017 at 8:10 PM, YongKang said:

It's possible and companies are actually getting profit from it haha

On anandtech they are talking about the 20k$ ballpark or 0,5$/GB. 

Perhaps you could add that to the post. 

 

Source: http://www.anandtech.com/show/11639/viking-ships-uhcsilo-ssds-50-tb-capacity-emlc-sas

 

[Juanitology]

On 7/15/2017 at 9:46 PM, leadeater said:

SSD cache wouldn't work, the amount required for the data rate would cost too much. The Netapp 3220 also only support 1TB per controller pair so 2TB and if you do that you severally limit the supported sizing as it is much less when doing that.

 

We do that on our production storage of Netapp 8040 6 node clusters for the SAS shelves for ESXi NFS volumes, 11 SSDs, but it doesn't make a huge difference. Our all flash shelves have such a performance difference it's not even worth comparing and that's to 15K RPM SAS cached with SSD and not 7.2K RPM SATA cached with SSD.

 

If we wanted to use flash pool for backup storage and keep using Netapp we would have to use 8060 nodes since 8040 maximum size possible for flash pool is 576TB, 86PB without it.

http://www.netapp.com/us/products/storage-systems/hybrid-flash-array/fas8000.aspx

 

Then there is the issue that SSD caching doesn't do anything for reads which is a big problem. We use Commvault for our backup application and that does synthetic fulls which entails reading the deduplicated data off of the backup storage to create the next full backup rather than reading it from the source. Then we also copy daily around 90TB to the other backup storage array in a different city which is another read operation, while other backup operations are running and synthetic fulls are being created as well.

 

The actual network traffic will be less due to deduplication, we generally write around 9TB-11TB to disk every day which is that 90TB before deduplication.

 

In front of those storage arrays is 6 HPE DL380 Gen9 servers with 4 NVMe write intensive SSDs and 2 SATA mixed use SSDs in them each, 3 servers in each city. All backup traffic flows through those servers and gets deduplicated before being stored on the backend storage. Only unique data is then stored on the backend storage and the performance requirements I talked about is for that data traffic which is way less that what hits the backup servers.

 

The deduplication allows us to store around 5.6PB on 450TB of used storage.

 

Since both production storage and backup storage is Netapp we can use Netapp snapmirror and snapvault technologies as well which we can do natively or use Commvault Intellisnap engine.

 

Cloud storage is dumb and slow and isn't great for using it as the first stage of backup storage, you can send copies of backups to the cloud but if you need performance to achieve your backup window the cloud isn't it. There are options but they are expensive and not always available in the location you are in.

 

Edit:

Oh and we also write data to tape for long term storage, we have 5 LTO-5 tape drives which you need to sustain 140MB/s to to avoid tape scrubbing, 700MB/s in total and when writing to tape you have to rehydrate/undedup the data. You can store deduplicated data on tape but it's best not to.

damn that is a massive system. sorry I did not grasp the scale of the system you're working with when I suggested ssd caching.

[leadeater]

1 hour ago, Terodius said:

damn that is a massive system. sorry I did not grasp the scale of the system you're working with when I suggested ssd caching.

Don't worry, you were still basically correct, just wouldn't work for us in our setup. If we moved to an object based storage setup (S3 or Swift) we could easily do what you suggested and it would work well. Two or four reasonably priced SSDs per server then scale out the number of servers to get the performance we need. We're just kind of stuck in the middle of needing the enterprise features of a storage array and being slightly too small to move to a similar infrastructure model as cloud providers.

[Bananasplit_00]

this is neat i guess, its nice to know we might have these in our computers in 5 years or so

[Juanitology]

1 hour ago, leadeater said:

Don't worry, you were still basically correct, just wouldn't work for us in our setup. If we moved to an object based storage setup (S3 or Swift) we could easily do what you suggested and it would work well. Two or four reasonably priced SSDs per server then scale out the number of servers to get the performance we need. We're just kind of stuck in the middle of needing the enterprise features of a storage array and being slightly too small to move to a similar infrastructure model as cloud providers.

Do you have a map/layout of your SAN?

[leadeater]

18 minutes ago, Terodius said:

Do you have a map/layout of your SAN?

Not really something I can post without taking info off it like serial numbers, IP addresses etc. Our setup isn't that amazing, each controller head (6) has two 10Gb connections to two different ToR switches and those ToR switched have two 40Gb connections to two different distribution switches. We do not stack our switches and all paths are active/passive so a hardware failure cannot cause over-subscription of networking bandwidth.

 

We then have two 10Gb/40Gb connections between cities for replication between the SANs, those links are used for all over inter datacenter traffic as well with VLANs segregating the traffic etc.

 

In essence it's just 6 servers (12 in total) with racks of disks attached to them, the special stuff is the software they run and the feature you get with them.

 

Netapp 8040 specs (per node)

 

CPU (Intel Xeon)                     Main Memory   NVRAM

1 x 64-bit 8-core 2.10 GHz

64 GB

16 GB