B.J.Blazkowicz brings the HDDpocalypse

[WereCatf]

Summary

 

Mechanical HDDs are slow, necessitating the use of various kinds of techniques in order to get loading-times down in games, including e.g. duplication of assets, causing the games to be larger than they technically need to be. Another common technique is the introduction of "chokepoints" in level-design, like e.g. making players wait a few moments in an elevator, making them crawl very slowly through some passages or similar things, so assets can be loaded from HDDs in the background. Alas, with the latest Xbox - and PlayStation - consoles ditching HDDs and moving to fast NVMe - storage, these techniques can be ditched and games can be designed around far faster access to assets, and that's exactly what MachineGames have announced they're doing for Wolfenstein 3 in a blog post.

 

 

Quotes

Quote

Jim Kjellin, Chief Technology Officer: One of the key items that's coming in the immediate future is the general availability of really fast storage. You've always been able to buy decently fast storage for your PC, but it will become standard in the next generation consoles.

 

This allows us to change how we think about game design. The historical issue of having to wait for loading to finish or artificially slowing the pacing before you can do major changes to the player experience will be a thing of the past.

 

John: I absolutely agree with Jim. With these super fast SSDs, we don't need to consider ways of entertaining the gamer while the levels load any longer! Allowing for bigger worlds and letting players jump around seamlessly within them are going to provide some exciting new experiences. I'm really looking forwards to that.

 

My thoughts

As a fan of the Wolfenstein - franchise and as a PC - gamer, I can't help but be rather excited to see how the game changes now that the constraints put on game - design by HDDs are removed. I haven't yet heard of any other PC - game ditching the HDD - based design, but with Wolfenstein 3 as the first or among the first, I predict we'll hear similar news for other games as well shortly. It looks like NVMe - drives may become a necessity for AAA - titles in modern gaming - PCs, finally pushing HDDs all the way down to the bottom, just for use as plain bulk - storage.

 

Sources

https://bethesda.net/en/article/6TiCksNgkXvYTWgAEuWHd0/nojs.html

Edited October 19, 2020 by WereCatf
Added an entirely 100% realistic image of Ol' Blazko going to war

[Mateyyy]

Mein leben!

Good to see that hardware will finally start to be properly utilised to (or at least closer to) its full potential in games, though I can't really see NVMe becoming a requirement just yet. If I had to guess, SATA SSDs are still far more common than NVMe/PCIe right now when it comes to PCs.

[WereCatf]

2 minutes ago, CircleTech said:

But there is one thing good about HDDs beyond just low cost storage - Data recovery. As of today, it is much easier to recover data from a dead HDD than SSD because the technology is at this point very well understood and the industry is heavily consolidated to a few players. With many SSD manufacturers using a variety of controller, caching, redundency, and storage geometry configurations, this is going to make standard data recovery a lot harder. 

True, but the counter-argument -- and a rather strong one at that -- is that you shouldn't rely on your data being possible to recover that way in the first place. Just simply keep backups of everything you count important.

[yian88]

Wohooo apps and games now will require at least SATA 3 SSD.

Thats what i want to see in 2020, and smaller new game footprint size, those 50-100GB games are sickening.

[PlayStation 2]

6 minutes ago, yian88 said:

and smaller new game footprint size, those 50-100GB games are sickening.

Get used to 100+GB games. They aren't gonna get smaller, even as advanced compression techniques come into play. Lossless audio and very high resolution texture data can only be compressed so much until it starts eating up CPU time that'd be useful for other things.

[Vishera]

100GB+ games and the prices of high capacity SSDs are valid reasons to use hard drives,

That's why the trend of SSDs as OS drives + HDDs for a game drive is popular.

[mariushm]

Flash is becoming so cheap and mass produced that I'm surprised WD or Seagate don't just add 120-250 GB of Flash for caching purposes, on drives where it would be profitable, like let's say 6-10 TB drives, making them 20-40$ more expensive. Then the problem becomes how games could hint to the drive that it should cache a folder or some files in a folder as they're read.

 

Game could load and send commands to the sata drive "I want this file and this file being read and cached in the background" and when it's idle (when you're in game menu or after loading a level for example), the hard drive's controller could sequentially read those big files and cache them in the flash storage.

 

Another potential solution would be just waiting for DDR5, which will most likely be made in 16 GB and 24 GB sticks (yeah ddr5 allows 1.5 GB chips, so you can use 8 chips  x 1.5 GB per chip = 12 GB stick) , so most people will have at least 32 GB. 

 

Right now you can buy 16 GB DDR4 for 50$, so you could just require at least 32 GB of ram to play the game... and the game could use 10-16 GB of that to cache in background all that it needs for the following half an hour or whatever.

 

[Brooksie359]

While I would like the idea of ssds being utilized for a game by design i would hope they wouldn't require the top of the line nvme ssd as that can be quite expensive especially with nvme slots being much more limited than sata ports. If its just sata ssd that is required I think it would be fine. 

[Vishera]

13 minutes ago, Brooksie359 said:

While I would like the idea of ssds being utilized for a game by design i would hope they wouldn't require the top of the line nvme ssd as that can be quite expensive especially with nvme slots being much more limited than sata ports. If its just sata ssd that is required I think it would be fine. 

Knowing how lazy game developers can get - just like SLI most of them will optimize the game to run on console hardware and then port to the PC.

But there is a silver lining - If the game engine does the optimization for storage then we will probably be fine. 

 

[porina]

27 minutes ago, Vishera said:

100GB+ games and the prices of high capacity SSDs are valid reasons to use hard drives,

That's why the trend of SSDs as OS drives + HDDs for a game drive is popular.

The SSD boot + HD for storage seems most popular in very budget builds, with typically a tiny OS SSD and a TB storage HD or so. For mid to higher end systems, I don't see SSD pricing as a barrier for game storage and up to TB/drive level is affordable.

 

There is the usual question, just how much space do you need at once? I used to install everything I own regardless if I played it or not, and estimate I need around 2TB or so. Since I uninstalled all the stuff I don't actually play, I have tons of free space.

 

It wont be a one size fits all solution for sure, but it would be nice at some point if more optimisations were available for SSD storage of games.

 

24 minutes ago, mariushm said:

Flash is becoming so cheap and mass produced that I'm surprised WD or Seagate don't just add 120-250 GB of Flash for caching purposes, on drives where it would be profitable, like let's say 6-10 TB drives, making them 20-40$ more expensive. Then the problem becomes how games could hint to the drive that it should cache a folder or some files in a folder as they're read.

I've kinda wondered why they don't use bigger flash caches. I've tried a couple of Seagate SSHDs in the past, and I can't say they feel any better than a HD. The newer one I had, had 8GB of flash. Maybe ok for fast booting Windows, but totally inadequate for anything else.

 

At a basic level, games wont need to do anything, and let the drive decide. Maybe on some hybrid of most recently used and most frequently accessed. So data that is used often and recently will be cached, but this would avoid infrequent assets like cutscenes for example. This doesn't get around the problem the first time anything is accessed will be slow. I think that's the fundamental problem. The underlying performance will always be HD level. If you want to raise the baseline to SSD level, you need a SSD.

 

The other end of the problem goes to my earlier comment, is this space really needed? Maybe if you want every game you own installed at once, but that's a lot of capacity. For other data storage uses, does a large cache make sense? 

 

24 minutes ago, mariushm said:

Right now you can buy 16 GB DDR4 for 50$, so you could just require at least 32 GB of ram to play the game... and the game could use 10-16 GB of that to cache in background all that it needs for the following half an hour or whatever.

PC gaming has been quite inclusive of lower end hardware and I don't see that going away. However there is quiet a lot of scaling that is possible. If you have higher end hardware, I hope games will make use of it.

[Fatih19]

2 hours ago, WereCatf said:

Wolfenstein

Hell yeah, dude!

[jagdtigger]

2 hours ago, handymanshandle said:

Get used to 100+GB games. They aren't gonna get smaller, even as advanced compression techniques come into play. Lossless audio and very high resolution texture data can only be compressed so much until it starts eating up CPU time that'd be useful for other things.

I dont think the CPU will be an issue, i mean you can use a PC with a fully encrypted drive and its barely noticeable. Plus we are at the point where 8 core chips and above becoming the norm so devs have more horsepower to work with.

[Fatih19]

1 minute ago, jagdtigger said:

I dont think the CPU will be an issue, i mean you can use a PC with a fully encrypted drive and its barely noticeable. Plus we are at the point where 8 core chips and above becoming the norm so devs have more horsepower to work with.

And what about that RTX I/O thing that Nvidia have?

[WereCatf]

1 minute ago, Fatih19 said:

And what about that RTX I/O thing that Nvidia have?

It supposedly handles that decompressing of assets in the GPU, so that'll decrease the load on the CPU even more. Plus the data doesn't pass through the CPU or system RAM at all and instead goes directly to the VRAM, so that reduces the load even further.

[jagdtigger]

1 minute ago, Fatih19 said:

And what about that RTX I/O thing that Nvidia have?

IDK, not following HW news that closely because i dont plan on upgrading anytime soon.......

[Loote]

So, what's the HDDpocalypse about?  Even before those changes QLC drives have become a good alternative for HDDs while the real use cases for them stay(backup space, offline storage that allows for random access when you need it, media storage). Are the small drives gamers buy a big part of the business?

 

If HDDs continue to increase capacities at an acceptable pace and keep better prices, or I guess even if they don't I see this as a opportunity to use tiered storage, no matter if your big capacity drives are QLC or HDD, you can buy a smaller high quality drive and push most used files there, though that probably depends how many people have hoarding tendencies and how often will we see things you don't have on your disk disappear(cloud providers closing etc.).

[WereCatf]

1 minute ago, Loote said:

So, what's the HDDpocalypse about?

Um, it reads in the summary. It's nothing to do with capacity.

[Loote]

1 minute ago, WereCatf said:

Um, it reads in the summary. It's nothing to do with capacity.

But HDDs are all about capacity, therefore things that aren't about it can't cause any sort of apocalypse for them.

[WereCatf]

Just now, Loote said:

But HDDs are all about capacity, therefore things that aren't about it can't cause any sort of apocalypse for them.

No, in the context of this news the capacity is irrelevant. The low access-times and the techniques game-devs have had to employ to work around those low access-times is the relevant thing.

[mariushm]

Hard drives are not used JUST for games, people use them for movies, lossless music, games that will NEVER require SSDs (because they can preload everything needed in ram or in the video card vram during level load), for backups, and other reasons. 

You also won't see services like backblaze or amazon glacier or google drive stop using mechanical drives anytime soon.

 

So no hddpocalypse anytime soon. 

 

Also, relax about that nvidia io thing... as long as AMD's gonna have a significant market share AND presence in consoles, few games are gonna package their textures and all that in a nvidia proprietary format, which would require the game to convert that stuff to something usable on AMD cards and consoles. 

IMHO no developer should use that proprietary nvidia thing, unless they make it open source or easily usable on amd and intel cards... with opencl or as a directx 12 extension it would be possible.  I guess there's gonna be develpers taking money from nVidia just to use that feature.

That takes me back to the S3 days when they invented S3TC (s3 texture compression) and made it available to other companies... then the amd and nvidia made their own texture compression schemes and directx then implemented compression... we're not using s3tc anymore. 

 

 

[WereCatf]

2 minutes ago, mariushm said:

IMHO no developer should use that proprietary nvidia thing, unless they make it open source or easily usable on amd and intel cards... with opencl or as a directx 12 extension it would be possible.  I guess there's gonna be develpers taking money from nVidia just to use that feature

It's far more likely that everyone will just use Microsoft's Direct Storage instead. As far as I know, that still doesn't prevent the GPU from doing the decompression of assets, either.

 

With the above in mind, I don't really know what RTX IO, specifically, brings to the table over Direct Storage, or if it's just some stupid marketing-scheme again.

[RejZoR]

I never understood duplication of assets. Why would you do that? Having one asset and parsing the same one multiple times is more efficient. It's literally a definition of instancing used for 3D graphics. Instead of rendering each same object as own entity, you render it once and just copy & paste it around. You still have to render all the final objects, but you have far less overhead. You'd think same applies to other things as well. Also not sure how multiple assets (duplicated) can be parsed faster when you need to parse them from same garbage slow HDD. It just makes no sense. And if you parse and store it in memory, first parsing is slower, all consecutive are much faster because they are accessed directly in RAM or VRAM. If i's not that way, then what the hell is this industry doing?!

[WereCatf]

1 minute ago, RejZoR said:

I never understood duplication of assets. Why would you do that? Having one asset and parsing the same one multiple times is more efficient. It's literally a definition of instancing used for 3D graphics. Instead of rendering each same object as own entity, you render it once and just copy & paste it around

What you are talking about requires you to already have that asset in memory, whereas duplication of assets at the storage-layer is before it's in memory -- two entirely different things. Duplication of assets at the storage-layer is done to reduce seek-times and improve amount of sequential reads, ergo being more HDD-friendly. HDDs SUCK at seek-times, but they're okay for sequential reads where they don't have to do constant seeks.

[Loote]

17 minutes ago, WereCatf said:

No, in the context of this news the capacity is irrelevant. The low access-times and the techniques game-devs have had to employ to work around those low access-times is the relevant thing.

I am picking a bone with 'HDDpocalypse' and the justification that HDDs becoming even worse primary storage for games is something with big influence over HDDs relevance in general. imo it's another nail, but nothing that's able to make a big dent on its own. Games are not everything.

[AldiPrayogi]

SSDs are still pretty expensive in my country. Really do hope that game developers give gamers the ability to choose which game assets to download. Like it'd be a waste to make everyone download 4K assets while I'm only playing on a 1080p monitor for example, it'd save some space for the SSD not to mention faster download time.