Moar storage - Seagate 24TB hard drives with HAMR technology to soon arrive in 2021.

[RejZoR]

1 minute ago, leadeater said:

Something being niche is defined by how many people have it, the deployment scale, not how useful or beneficial it could be to people. SSD caching and SSHD (these are different) is certainly a niche thing, very few actually do it.

It's a niche thing because it was designed by a moron, delivered by an idiot and never improved by a retard. Of course no one bought that hot garbage and it never became popular. Between absolutely useless SSD boot drives everyone was running and shuffling games and apps back and forth like idiots, SSHD's would do all this automatically while providing massive capacity at minimal premium. Instead people kept using dumb SSD boot drive method and some still do today. It's just beyond baffling.

 

SSHD's could easily be the next big thing yet vendors themselves fucked it up so hard it just never took off. And they would still be relevant TODAY if they did it right and people got used to them. Instead because of dumb execution about basically everything around SSHD's, they just flopped so hard no one even remembers them anymore. You can't just say "it's a niche thing" when it never even had a chance to take off. Not because the idea itself is bad, it's because execution was just straight out moronic. I mean 8GB cache for god sake, back when SSHD's with this crappy cache were somewhat relevant I had 2TB HDD paired with 32GB SSD cache and software was doing the caching. I was mostly playing Natural Selection 2 back then which is a +10GB game. Garbage SSHD's with pathetic 8GB SSD cache couldn't even cache this entire game. Yet my 32GB one could easily cache SEVERAL games as well as all the apps I used and also OS. Boot to desktop was as fast as now that I'm on pure 2TB SSD. Apps launched just as fast as now. I've had game launch speeds the same as now. It was so good I'm seriously thinking of doing it again as I bought 8TB HDD for data hoarding some time ago and it's hardly being used where 2TB SSD is sort of running out of space as I'm holding it back a bit. 8TB HDD paired with 2TB SSD would give me insane 8TB of space with 2TB SSD cache for everything I'd ever need. Good luck buying 8TB SSD without selling a kidney... Niche thing my bottom.

[leadeater]

22 minutes ago, RejZoR said:

It's a niche thing because it was designed by a moron, delivered by an idiot and never improved by a retard. Of course no one bought that hot garbage and it never became popular. Between absolutely useless SSD boot drives everyone was running and shuffling games and apps back and forth like idiots, SSHD's would do all this automatically while providing massive capacity at minimal premium. Instead people kept using dumb SSD boot drive method and some still do today. It's just beyond baffling.

 

SSHD's could easily be the next big thing yet vendors themselves fucked it up so hard it just never took off. And they would still be relevant TODAY if they did it right and people got used to them. Instead because of dumb execution about basically everything around SSHD's, they just flopped so hard no one even remembers them anymore. You can't just say "it's a niche thing" when it never even had a chance to take off. Not because the idea itself is bad, it's because execution was just straight out moronic. I mean 8GB cache for god sake, back when SSHD's with this crappy cache were somewhat relevant I had 2TB HDD paired with 32GB SSD cache and software was doing the caching. I was mostly playing Natural Selection 2 back then which is a +10GB game. Garbage SSHD's with pathetic 8GB SSD cache couldn't even cache this entire game. Yet my 32GB one could easily cache SEVERAL games as well as all the apps I used and also OS. Boot to desktop was as fast as now that I'm on pure 2TB SSD. Apps launched just as fast as now. I've had game launch speeds the same as now. It was so good I'm seriously thinking of doing it again as I bought 8TB HDD for data hoarding some time ago and it's hardly being used where 2TB SSD is sort of running out of space as I'm holding it back a bit. 8TB HDD paired with 2TB SSD would give me insane 8TB of space with 2TB SSD cache for everything I'd ever need. Good luck buying 8TB SSD without selling a kidney... Niche thing my bottom.

No matter what you say it's still niche, that's exactly what it means. Even though software exist to do it, and for ages, very few use it. Also fyi it's very much easier to do it in software at the OS layer than it is to do at the SSHD controller layer, that's why it was dropped and never developed. SSDs are getting cheaper at a rate faster than SSHD will ever be truly useful to the masses and very few people actually need more than 2TB so there is no reason to develop a solution for the 0.01% when they can utilize already existing options like Primocache.

[RejZoR]

Eh forget it. You're still not getting it. It would not be a niche thing if moronic SSD boot drive method wasn't so popular for some dumb reason and if they've done them right with larger cache. Having massive storage with near SSD speeds for real world use at fraction of a cost of full blown SSD, I can assure you SSHD's would took of and become the mainstream. Instead they were sort of alright in benchmarks, sort of managed to make system boot faster and that was about it because they just didn't have the SSD cache capacity to be useful for anything else. And SSHD being a single unit with no need for software or any fiddling, just plug and play it would work brilliantly. PrimoCache is great and offers better control for me as advanced user, but it can be fiddly and overwhelming for casual users. And it's casual users who would benefit from SSHD's the most. Just stick it in and you have big capacity with excellent day to day use speeds.

 

Besides, I don't expect anyone who hasn't used a properly functioning hybrid storage to understand any of it. I've seen it with my own eyes, used it for years and it was amazing. Only went with full SSD because of noise as I sleep in the same room as I have PC and it's on 24/7. And because I could (mind you that Samsung 850 Pro 2TB SSD was over 800€ when I bought it, hybrid setup would cost 1/4th and probably perform very close). I'm actually again thinking of pairing my 8TB HDD with 2TB SSD again... Just because I have both at hand and that 2TB SSD is getting a bit crowded...

[leadeater]

2 minutes ago, RejZoR said:

It would not be a niche thing if moronic SSD boot drive method wasn't so popular for some dumb reason and if they've done them right with larger cache. Having massive storage with near SSD speeds for real world use at fraction of a cost of full blown SSD, I can assure you SSHD's would took of and become the mainstream

It's not moronic because it actually works. Like I said SSHD development stopped because it's just not as easy to implement or effective as you want or think it could be. Primocache works so well because it's software installed at the OS layer and thus has access to the filesystem layout and other information that an SSHD would never have access to doing it on the disk controller and thus it will never be effective enough to warrant the development cost for something that will always be rather bad.

 

HDD manufacturers stopped SSHD development for this reason, not because such a thing isn't a good idea it's just not technically feasible to do it effectively. Both are NAND manufacturers so there is no conflict of interest either. If there was real merit in pursuing it they would have, they didn't because there isn't.

 

Something being a good idea is totally different to it actually being possible. So if you want SSD caching use Primocache, or something else, and promote those. SSHD is not happening.

[leadeater]

8 minutes ago, RejZoR said:

Besides, I don't expect anyone who hasn't used a properly functioning hybrid storage to understand any of it.

Well for you information I have such a thing at home and at work. I guess the difference is I put realistic possibilities before idealism. 

[CarlBar]

Yeah i might use up several TB of data like it's nothing, so might a fair few other here, but your average PC user barely has any games installed if any at all and they watch/listen to media on their tablets or smart tv's. They just don't need more than a couple of TB of storage for what they do.Even thats often overkill. Some cna get by on a 1Tb or even 512GB SSD no problem. The thought of the latter gives me anxiety.

[GDRRiley]

21 hours ago, Yebi said:

I'm definitely with Linus on this one (referring to that old WAN show clip about 20TB drives). The imbalance between capacity and speed is getting too big for HDDs this large to be a good choice. You don't want to spend two days rebuilding a failed drive

if I'm backblaze why would I care? they build in 3 levels of redundancy on drives. the density is key for places like them

[AndreiArgeanu]

8 hours ago, Brooksie359 said:

I think part of the reason why people don't choose a harddrive plus ssd solution is that ssd prices are so cheap now that you really aren't saving much money until you get to large capacity drives. I mean you can get a 1 tb ssd for super cheap

Lots of people I know, including myself use a hard drive and ssd combo, and you are saving money by doing so. You can get a decent 250GB ssd + 2TB hard drive for a cheaper/similar price to a decent 1TB ssd. Not to mention that 1TB of storage can be filled fairly easily considering how much space modern games take, so after formatting, a windows 10 installation, a few programs and some games the ssd will top up. Of course 1TB isn't by any means not enough, it's manageable but it's not the most ideal solution.  While yes, SSD's have gotten so much cheaper in the past years they're not even near the price of an HDD.

[Doobeedoo]

6 hours ago, RejZoR said:

It's not a niche thing when you want a fast capacity drive. 2TB SSD's are still very expensive for majority of people. Where SSHD's could deliver 5x capacity at near SSD speeds for basically a price of a slightly more expensive HDD. It's still just incomparable. And with larger SSD cache on board they'd actually perform, because with garbage 8GB cache they had, it was terrible when they were new. At that time I was running a hybrid setup via software solution and a 32GB SSD cache. Which I upgraded to 256GB some time later. I've had OS, all the daily used apps and even massive games (several of them!) cached in it. Good luck ever doing that with 8GB cache without swapping data in in and hurtin performance. I don't believe it that SSHD's are pointless. If only they weren't designed by morons and never fixed accordingly later. They released dumb 8GB cache design, it didn't work all that well and people weren't convinced and they just scraped them entirely. Hell, even 128GB cache that would cost just additional 50€ at worst and would perform so much better than absolutely worthless 8GB cache...

Yeah by being niche it is to many. Many barely know difference between HDD and SSD let alone know doing hybrid caching storage now. They never made multi TB HDD with much larger SSD cache, so that bridge-gap was done quickly. Aside from hybrid solution from user. But yeah, very few do it.

[RejZoR]

It's almost as if I wasn't explaining why it became a niche. Or shall I say why it remained niche... And people damn well know what SSD is. And have for quite some years now. Even the most clueless ones. They may not know the specifics of guts inside, but everyone knows HDD's are slow and SSD's are not.

 

Also saying it's not possible to do it in hardware is load of BS. It's block level caching, basically same as used by PrimoCache. You just do it on the drive instead in OS. Only difference is that you can't fiddle with it like in PrimoCache to have additional functionality like deferred writing or memory caching. It's same crap, here driver is intercepting the read requests, there it would be the SSHD controller saying "hold on, what you want to access is actually stored on SSD part, let me fetch that instead". If not it'll just read of the platters. For majority of users, it would deliver day and night performance at fraction of a cost and at insane capacities. Power users might prefer PrimoCache. Or just going full SSD if you're loaded and it's no problem running 2x 4TB SATA drives in RAID0 for maximum speed and capacity. Coz finding one in M.2 format is a bitch, just like it was when I was looking for 2TB SSD. They simply didn't exist in 2TB format. It's only been recently and even that only by few makers. The rest only goes up to 1TB for some stupid reason... So, good luck storing your massive games there...

[StDragon]

10 hours ago, leadeater said:

Correct, induvial disks are known as OSDs and these get grouped in to Placement Groups (PG). The CRUSH rules and Pools configuration allow to you define how data is distributed across the cluster, ours is done at the host level so any data copies or chunks of parity are spread across servers.

 

Data objects are placed in to PGs and the PG chooses the OSD with the least data on it (typically).

 

The cluster builds a map of the hardware and then you can create logical partitions of that, multiple as you can have different pools with different settings using different CRUSH rules, and then data objects are stored based on these rules and configurations. If hardware fails the cluster starts moving data around to ensure the data resiliency is retained, there is no waiting for hardware to be replaced. This does mean you cannot actually fill the entire cluster as you require empty space for those data management operations.

So basically a data pool fabric that's spread throughout the enterprise...

 

What terrifies me about this concept is from an ops perspective. How exactly would you perform disaster recovery against a portion of a hardware failure? Or, data injection from backup, and what, or where, to physically inject the data to pre-seed for the least amount of downtime/replication for the end-user?

 

Basically, I see the same issues of DFS plaguing this solution as well. I don't blame the implementation per se, but more of the philosophy of being non-centric in the first place. IMHO, you're trading one problem for another.

 

I'm assuming there's a whole set of management tools and physical methodologies implemented in regards to planning for DR? I'd worry too much data spread-out geographically could get unwieldy.

[leadeater]

6 hours ago, RejZoR said:

Also saying it's not possible to do it in hardware is load of BS. It's block level caching, basically same as used by PrimoCache. You just do it on the drive instead in OS.

No it is not BS, tell me which blocks are for what and relate to what without the filesystem table? Can't can you? Yep you're doomed to do caching to any degree of efficiency without that.

 

Doing it on the drive is not the same as PrimoCache.

 

6 hours ago, RejZoR said:

It's same crap, here driver is intercepting the read requests

There is no driver intercepting anything if you do it on the disk controller rather than in OS software.

 

Just because you're convinced it's possible with zero real technical issues doesn't actually mean that is the case. You can either believe the issues I have point out or sit and rage at something not happening because you are convinced it can be despite the industry and manufacturers not doing it even though it would be in their best interest and business benefit to do it.

 

And FYI a proper caching solution that is working efficiently does not need to nor will it cache all 10GB of a game because never is all 10GB of the game files required. Doing a bad cache implementation risks cache thrashing which will wear out the NAND flash extremely fast and then SSHD would get a reputation of being unreliable garbage and not used either.

 

6 hours ago, RejZoR said:

For majority of users

You mean the majority that don't even have more than 1TB of storage usage?

[leadeater]

3 hours ago, StDragon said:

So basically a data pool fabric that's spread throughout the enterprise...

No you build a cluster with servers for this purpose, exactly the same as Backlaze does but they use their own software defined storage solution.

 

3 hours ago, StDragon said:

 

Basically, I see the same issues of DFS plaguing this solution as well.

What issues with DFS? Or do you specifically mean DFS-R? DFS-R is replication at the file level and the only real issue with that is it can't handle file locks or if a file is write locked changes don't replicate until the file handle is closed. DFS-R isn't widely used anyway. DFS Namespaces are however and I can't say there is any issues with that, using that is a huge benefit as you never have to change share mount paths even when you move the share to a different server or volume and path on the storage array.

 

Not that DFS is anything like Ceph at all mind you.

[StDragon]

53 minutes ago, leadeater said:

No you build a cluster with servers for this purpose, exactly the same as Backlaze does but they use their own software defined storage solution.

 

What issues with DFS? Or do you specifically mean DFS-R? DFS-R is replication at the file level and the only real issue with that is it can't handle file locks or if a file is write locked changes don't replicate until the file handle is closed. DFS-R isn't widely used anyway. DFS Namespaces are however and I can't say there is any issues with that, using that is a huge benefit as you never have to change share mount paths even when you move the share to a different server or volume and path on the storage array.

 

Not that DFS is anything like Ceph at all mind you.

DFS-R. FYI, it replicates at the file block level instead of file like the old obsolete FRS.

 

Anyways, you're right about op-locks between servers. Though it's fine if multiple people are working from the same source (site). But I digress. It's the same issue that effects cloud based file solutions too. They have their place, but many industry needs such a GIS need to keep that data (and lots of it) local.

[leadeater]

3 minutes ago, StDragon said:

DFS-R. FYI, it replicates at the file block level instead of file like the old obsolete FRS.

It can use remote differential compression yes but it still monitors at the file level and thus has the issue of files being write locked. I know about that issue because I've been hit with it and done a bunch of tests to figure out that is the case. So in my view I classify it as file level replication because that is the mechanism to actually monitor changes rather than at the block level like is the case on something like a Netapp FAS.

[Maticks]

this looks really interesting so if there is a laser doing the read and writing does that mean you can't have a head crash anymore?

nothing is physically touching the plater anymore, i wonder...

 

Also might mean a repair to get the data off the disk is nothing more than switching out some laser.

this is interesting technology for sure. half the issue is keeping the heads seperated when replacing them.

 

Looking forward for a LTT tear down and see how this works.

certainly going to add a lot more cost.

[JediFragger]

Stop.....HAMR time.

[leadeater]

12 minutes ago, Maticks said:

nothing is physically touching the plater anymore, i wonder...

It wasn't before either, the heads sit above the platter and I believe this height is unchanged.

[Maticks]

14 minutes ago, JediFragger said:

Stop.....HAMR time.

wonder if that will be the offical S.M.A.R.T response at failure.

[Brooksie359]

16 hours ago, AndreiArgeanu said:

Lots of people I know, including myself use a hard drive and ssd combo, and you are saving money by doing so. You can get a decent 250GB ssd + 2TB hard drive for a cheaper/similar price to a decent 1TB ssd. Not to mention that 1TB of storage can be filled fairly easily considering how much space modern games take, so after formatting, a windows 10 installation, a few programs and some games the ssd will top up. Of course 1TB isn't by any means not enough, it's manageable but it's not the most ideal solution.  While yes, SSD's have gotten so much cheaper in the past years they're not even near the price of an HDD.

Let's say I only need 2 tb of storage. Sure I could get a 256 gb ssd and a 2 tb had for like 100 bucks but for 80 bucks more I can get a 2 tb nvme ssd. I would pay the 80 bucks more basically every time. As for 1 tb of storage even if you did a 256 gb ssd and a 1 tb harddrive it would still be not a whole lot more expensive to simply get a 1tb ssd. Also larger capacity ssds are faster than the 256 gb counterparts. Sure if you need 4tb of storage then yeah a cashe solution might be worth it but honestly most people won't be bothered with an had unless they need alot of storage. 

[RejZoR]

5 hours ago, Brooksie359 said:

Let's say I only need 2 tb of storage. Sure I could get a 256 gb ssd and a 2 tb had for like 100 bucks but for 80 bucks more I can get a 2 tb nvme ssd. I would pay the 80 bucks more basically every time. As for 1 tb of storage even if you did a 256 gb ssd and a 1 tb harddrive it would still be not a whole lot more expensive to simply get a 1tb ssd. Also larger capacity ssds are faster than the 256 gb counterparts. Sure if you need 4tb of storage then yeah a cashe solution might be worth it but honestly most people won't be bothered with an had unless they need alot of storage. 

And what will you do when you need 4TB? 6TB? 8TB? 10TB? Are you going to spend 25.000€ on an SSD of that size? Or some 600€ for same capacity with near SSD speeds? 2TB is a hard limit on M.2 and you just can't have more. Unless you want to have bunch of split storage with several sticks which sucks.

[LAwLz]

45 minutes ago, RejZoR said:

And what will you do when you need 4TB? 6TB? 8TB? 10TB? Are you going to spend 25.000€ on an SSD of that size? Or some 600€ for same capacity with near SSD speeds? 2TB is a hard limit on M.2 and you just can't have more. Unless you want to have bunch of split storage with several sticks which sucks.

I don't see what benefit you seem to see from using an SSD cache, and I don't think the other people in this thread sees it either.

If you need let's say 10TB of storage then you won't be needing 10TB of high speed storage. Most of that will be bulk storage which can be saved on an HDD. you might need 1TB of fast storage and 9TB of bulk storage.

 

Nobody is saying "HDDs are dead, just get an SSD for all your storage". People, at least I am, are saying "if you need high speed storage and bulk storage keep them separate. Buy a 1TB SSD if you need 1TB of high speed storage, and buy a 4TB HDD if you need 4TB of bulk storage. Don't buy a 0,5TB SSD and a 4TB HDD because you don't really gain anything from it". Or if you only want high speed storage, get a big SSD.

 

What's important to remember is that an SSD cache with an HDD does not increase performance. You still get the fast speeds on the SSD only, and the slow speeds on the HDD only. An SSD cache does not magically turn your HDD into an SSD. At least not for reads. I think I saw in some other thread that you get like 95%+ cache hits so therefore you almost always get the high speeds. Do you know what that means? It means that your HDD is barely used and you might have well have gone with the SSD as your only storage. Or maybe the SSD as its own separate drive where you got all your programs installed, and then a separate drive and partition for your HDD where you save your bulk storage like movies, music, and other things that do not need high speed.

 

 

Also, 2TB is not a hard limit for M.2. Not only is it not a hard limit in the technical specs, we even have larger than 2TB drives on the market.

 

 

Just buy an SSD of the size you need, and one or more HDDs of the size you need. Then use them as separate drives. Install Windows, programs and games on the SSD and they will always be fast. Store your movies and other media files on your HDD.

Don't try and save a couple of dollars by buying an SSD smaller than your needs and a HDD to cover for it. In the end you will not save that much money (especially not if you keep recommending expensive caching software since you don't like StoreMI or RST like you do) and you will get inconsistent performance since you will sometimes get cache misses.

 

The only time an SSD cache makes sense for home users, in my eyes, is if you want really fast write speeds to your bulk storage. Like if you are recording very high bit rate video for very long periods of time. That is an extremely niche use case. 

[Brooksie359]

44 minutes ago, RejZoR said:

And what will you do when you need 4TB? 6TB? 8TB? 10TB? Are you going to spend 25.000€ on an SSD of that size? Or some 600€ for same capacity with near SSD speeds? 2TB is a hard limit on M.2 and you just can't have more. Unless you want to have bunch of split storage with several sticks which sucks.

I already said that until you get to larger capacity hhds don't make sense. Obviously at 4t and above you would have to think about if you really wanted to pay alot for 2 tb sata ssds or a mix of nvme and sata ssds. Currently I have a 2 tb nvme ssd and a 2 tb sata ssd which amounts to 400 in price. Sure it is definitely much more expensive than having say 2x 2tb hhds and a ssd for cashing and at that point one would have to consider if it is personally worth it to them. I think at lower capacity though hhds don't make a tone of sense. 

[Sugadaddy]

Just thinking of the noise this thing will make gives me a headache.

[leadeater]

1 hour ago, LAwLz said:

-snip-

Lets also not forget that the new generation and the current/old generation of consoles don't have more than 1TB (usable less than that even) and games are still the most common large storage users for most people, far more common than any other usage.

 

Yes games are getting bigger, not all of them are huge though, and there is pending technology changes that might actually reduce the size of game installations so who knows what the data growth is going to be like. I also understand the new generation of consoles have/will have expandable storage, computers already have this and if you must have it under a single drive letter for what ever reason you can even do that in a multitude of different ways.

 

But there is also another side to this, when it comes to games how many of them you own do you actually play, not many at all. If you live in a country and have access to even half decent internet even the largest games do not that that long to download so deleting games you do not currently play is a perfectly legitimate data management option and cheaper than buying a larger SSD, buying an HDD, or setting up SSD caching. Simply have a look at what you do and do not need and get rid of the do nots, the cheapest option of everything.