DirectX 12, Direct Storage 1.1 is coming this year? to take your SSDs!

[SimplyChunk]

Finally...a proper use for realizable BAR

[cmndr]

On 10/14/2022 at 12:43 PM, CHICKSLAYA said:

This is partially why I have started suggesting PCIE4 SSD's to people for new builds

This is fine if it's about the same cost but doesn't really make a difference since this tech is on the same release schedule as Duke Nukem 4Ever and Half Life 3.

[LAwLz]

14 hours ago, cmndr said:

This is fine if it's about the same cost but doesn't really make a difference since this tech is on the same release schedule as Duke Nukem 4Ever and Half Life 3.

And there is far more to an SSD's performance than just what interface it uses.

There are plenty of PCIe 3.0 drives that performs better than a lot of PCIe 4.0 drives. 

[StDragon]

1 hour ago, LAwLz said:

And there is far more to an SSD's performance than just what interface it uses.

There are plenty of PCIe 3.0 drives that performs better than a lot of PCIe 4.0 drives. 

And then there's Dell's SSD classification (currently 10 - 50 on last page). Always thought this was stupid.

Basically, don't by some cheap no-name brand SSD.

[Quackers101]

when PCIe 4.0 to PCIe 5.0 drives be more expensive, but only for the PCIe part and the SSD be caught lacking. oof.

When you get an 100 dollar SSD, but the 30 dollar outperforms it.

[Stahlmann]

All nice and well, but we have yet to see ANY game that uses direct storage in any meaningful way (or in any way afaik).

[Stahlmann]

On 10/16/2022 at 2:52 AM, SimplyChunk said:

Finally...a proper use for realizable BAR

Resizeable BAR has nothing to do with direct storage.

 

Direct storage is an interface between VRAM and storage (SSDs / HDDs)

Resizable BAR is an interface between your CPU and VRAM.

[starsmine]

4 hours ago, Stahlmann said:

All nice and well, but we have yet to see ANY game that uses direct storage in any meaningful way (or in any way afaik).

From the outside, I see that a lot of supporting the XSS makes deves go, why not support PS4 pro/XBONEX which then means they have negative reasons to build games around things like direct storage. 

Forspoken being the first PC game to support it. 

Honestly supporting Direct storage on PC at this point is more limiting your audience then games like Crisis on launch did. If a game expects to load something in .x ms and it cant its going to stutter and hitch something wild. Crisis you could always play on low.

 

PS5 went so hard on Direct storage with having the ASIC decompression chip on an insanely fast PCIe gen 4 drive, that you cant go multiplatform and support this kind of thing. For third party devs, its just going to be easier to go for install base unless subsidized. 

Square with Forspoken is just cause Square always has a division just to push tech in insane ways. Thats why they are one of the rare publishers that build their own engines. and are being subsidized by sony to build it.

[Franck]

On 10/14/2022 at 11:21 AM, Quackers101 said:

but has to be added and maybe updated in most directx 12 titles.

Yes and fortunately it's not difficult to change. It's not a new thing though. We already have it in preview in 1.0 but it's more code to write and i didn't see that performance uplift. although i haven't try on Windows 11 but with other performance increase of Windows 11 for C/C++/C# api call i would doubt it would be marginally faster. It does seems to be more straight forward as in a single line of code call with the 1.1 version.

[ewitte]

49 minutes ago, starsmine said:

Honestly supporting Direct storage on PC at this point is more limiting your audience then games like Crisis on launch did. If a game expects to load something in .x ms and it cant its going to stutter and hitch something wild. Crisis you could always play on low.

 

That is why it would be added as a quality-of-life feature.  You can wait for the game to load 30 seconds or you can wait 5.  Plus, not having as high view distances, possibly lower resolution textures, etc.  

[tim0901]

On 10/15/2022 at 6:24 PM, leadeater said:

Developers may have chosen not to adopt something if the benefits aren't good enough for the effort. Adoption takes time but I suspect it's the former, I don't think 1.0 really had much value personally.

The intention with 1.0 was to allow developers to start integrating it into their game engines. DirectStorage isn't the kind of feature you're going to be able to add into an existing engine over a weekend - while yes in theory any DX12 title is compatible, it's going to require some serious restructuring of how things work under the hood in order to be used effectively. Being able to do this work earlier in the development cycle of upcoming games is a big deal.

[leadeater]

1 hour ago, tim0901 said:

The intention with 1.0 was to allow developers to start integrating it into their game engines.

Having any defined standard allows this, doesn't mean anyone is going to choose to do it. UE5 I think committed from the get go although I would have to double check, but also just because it's there in UE5 (assuming also usable) doesn't mean game developers themselves will also use it.

 

1 hour ago, tim0901 said:

Being able to do this work earlier in the development cycle of upcoming games is a big deal.

So is prioritizing work effort and development costs and evaluating resourcing utilization against benefits. If there are other things to do with tangible benefits and customer requests those are more likely to come first.

 

That's the area DirectStorage 1.0 sits in, it doesn't actually allow any of the fundamental changes touted by DirectStorage.

 

Add a new never used feature in to the development pipeline of the game engine or game along with all the QA/testing required for it just to lower CPU utilization a little bit which isn't even a problem or don't do it. Now you as the development production manager what is your choice?

 

Not saying DirectStorage 1.0 is bad or useless but it's pretty fair to see why it's not been used much and few games coming with it as yet, far as we know.

[GoodBytes]

49 minutes ago, leadeater said:

Not saying DirectStorage 1.0 is bad or useless but it's pretty fair to see why it's not been used much and few games coming with it as yet, far as we know.

We need to wait for game engines like Unreal to support it.

And even then, publishers looks at the lower common denominator, keep that in mind.

And that is assuming a game can run on BOTH, system with DirectStorage and those who don't support it fine.

 

Looking at DX10, 11 and 12, it took a VERY long time for games to actually support it... and I mean use it... not: "some special effect in some room in some part of the game, just to say "DirectX 10!" or whatever was used on the box."

 

[tim0901]

3 hours ago, leadeater said:

So is prioritizing work effort and development costs and evaluating resourcing utilization against benefits. If there are other things to do with tangible benefits and customer requests those are more likely to come first.

 

That's the area DirectStorage 1.0 sits in, it doesn't actually allow any of the fundamental changes touted by DirectStorage.

 

Add a new never used feature in to the development pipeline of the game engine or game along with all the QA/testing required for it just to lower CPU utilization a little bit which isn't even a problem or don't do it. Now you as the development production manager what is your choice?

Depends what you're working on.

 

If you're developing next year's FIFA, no it's probably not worth it. If you're developing a AAA title that's releasing in a couple of years time - say GTA 6 - yes absolutely it's something you'll want, especially given it has already proven to be a very valuable feature on the consoles - DirectStorage has existed on the Series X/S since launch, so calling it a "never used feature" is not entirely accurate.

 

And it doesn't just lower CPU usage - the loading times are faster as well. For many games it's going to be obvious from early on that it'll be helpful - anything that relies on asset streaming for example.

 

3 hours ago, leadeater said:

Not saying DirectStorage 1.0 is bad or useless but it's pretty fair to see why it's not been used much and few games coming with it as yet, far as we know.

No, we haven't seen games shipping with 1.0. Because without the GPU decompression it provided little-to-no benefit to the end user. Games will either ship with 1.1, or not ship with it at all. But my point is that that was never its purpose - Microsoft never intended for games to ship with it.

 

Let's reframe it slightly: if we call DirectStorage 1.0 "DirectStorage Developer Preview" does that make its intended purpose clearer? It allowed developers who were already interested in the technology to get a head-start on implementing it into their games. The same way that Apple has their iOS Developer Beta releases every year. So that when the final release arrives, you've done the majority of the work already.

[leadeater]

21 minutes ago, tim0901 said:

DirectStorage has existed on the Series X/S since launch, so calling it a "never used feature" is not entirely accurate.

Yep but note DS 1.0 isn't equivalent to that, only 1.1 is. That's why I said it could easily be chosen to skip 1.0 because it doesn't allow anything that DS is touted for or anything the consoles can do. DS 1.0 really just lowered CPU overhead to do the same task as before without changing how games can be designed, unlike with the consoles.

 

21 minutes ago, tim0901 said:

But my point is that that was never its purpose - Microsoft never intended for games to ship with it.

Exactly, hence my reasoning behind why some may have chosen to do nothing at all and wait for 1.1. I honestly don't think implementing 1.0 gives much of a head start.

 

But like I said, I'm sure UE5 committed to doing it but I can't be bothered to check as I don't think it matters much. Who did what and why doesn't change much for DS 1.0 really, this is just one possible reason why we haven't seen it.

[Sauron]

Cool but I don't think it's that impactful to just have the level load faster. We're already at the point where open worlds have virtually no loading screens anyway.

[Mihle]

5 hours ago, Sauron said:

Cool but I don't think it's that impactful to just have the level load faster. We're already at the point where open worlds have virtually no loading screens anyway.

That is just a nice side effect, the main benefit is that you can have higher res textures and higher polygon models more easier while using less off the CPU.

[Quackers101]

12 hours ago, Sauron said:

Cool but I don't think it's that impactful to just have the level load faster. We're already at the point where open worlds have virtually no loading screens anyway.

If you didn't see what it was in the OP, although very specific scene (likely not in real world performance).

from 100% CPU usage to 15%, faster time, and so on. Something like AC (assassin creed) could benefit a lot from, for textures/details + more space for having CPU demanding AI entities in it's world or more complex AI.

[Quackers101]

rumor or statement, directstorage 1.1 might come this week or next week?

When you see new driver and windows releases that could include this.

Then the games have to support it, and how well it supports this feature (mostly newer titles with this included should maybe run better).

[GoodBytes]

43 minutes ago, Quackers101 said:

rumor or statement, directstorage 1.1 might come this week or next week?

When you see new driver and windows releases that could include this.

Then the games have to support it, and how well it supports this feature (mostly newer titles with this included should maybe run better).

we are still waiting for game consoles games to support it... (beside R&C on the PS5) 

[Kisai]

Consider that nvidia's latest game driver is completely unstable. It would not surprise me if this was related. Between the SYSTEM_PTE_MISUSE error I was getting and several Unity programs flipping out and not loading.

 

My opinion is that I feel this is easy to implement with a backwards-compatible mode. If a game doesn't support DS1.1 then it will just decompress it in software, and you lose out on the performance gain. So the only thing keeping game developers from just doing that is probably the licensing.

https://github.com/NVIDIA/nvcomp

Quote

From version 2.3 onwards, the compression / decompression source code will not be released. We'll continue to maintain this Github for documentation and code sample purposes.

...

Known issues

• Cascaded, GDeflate, zStandard, Deflate and Bitcomp decompressors can only operate on valid input data (data that was compressed using the same compressor). Other decompressors can sometimes detect errors in the compressed stream.

So, no source code available, and nvidia's relies on CUDA, so on AMD and Intel.

 

[porina]

8 hours ago, Quackers101 said:

rumor or statement, directstorage 1.1 might come this week or next week?

According to MS it was out a few days ago:

https://devblogs.microsoft.com/directx/directstorage-1-1-now-available/

 

4 hours ago, Kisai said:

My opinion is that I feel this is easy to implement with a backwards-compatible mode. If a game doesn't support DS1.1 then it will just decompress it in software, and you lose out on the performance gain. So the only thing keeping game developers from just doing that is probably the licensing.

What license? Would have thought it'll be covered if you're using DirectX already.

 

On that note, I wonder if it can be used with Vulkan?

 

4 hours ago, Kisai said:

So, no source code available, and nvidia's relies on CUDA, so on AMD and Intel.

Presumably AMD/Intel will either have their own optimised equivalents, or fall back to generic implementation. The potential compatibility problems is listed as a known bug so presumably it is being looked at. We're probably still some way off major game support so there's time to resolve things like this.

[Kisai]

6 hours ago, porina said:

According to MS it was out a few days ago:

https://devblogs.microsoft.com/directx/directstorage-1-1-now-available/

 

What license? Would have thought it'll be covered if you're using DirectX already.

The GDeflate is basically GPU-dependent, since an encoder on an AMD or Intel part would produce a different result from the NVIDIA part because the code is different. 

 

Now that said, If the ONLY algorithm used is GDeflate, that's basically a parallelized PNG. Deflate (LZ77) used in the traditional sense is the original Zip file algorithm. So software encoders and decoders exist for that already, its's called Zlib. If you read the MS Blog they even describe how it works.

 

Quote

Knowledge of how GDeflate works can inform decisions about appropriate stream sizes. First, GDeflate splits the uncompressed stream into 64 KiB-sized tiles.  Each tile is compressed separately. This provides the first level of parallelism – for CPU decompression, each tile can be decompressed by a different thread, and for GPU decompression, each tile can be decompressed by a single thread group.  For the second level of parallelism, GDeflate arranges the data within a tile so that multiple lanes within a thread group can work in parallel on decompressing that tile.

 

64KiB tiles is the same maximum chunk size of Zlib. What's functionally different here is that it's basically the equivalent of "cut the raw file into 64KiB" chunks and then Deflate those individually. Near the end of the post they say it can be done on the CPU

Quote

Controlling CPU Decompression

When DirectStorage performs GDeflate decompression on the CPU it uses its built-in decompression system. This uses a Windows threadpool to support multi-threaded CPU decompression. The NumBuiltInCpuDecompressionThreads field in DSTORAGE_CONFIGURATION can be used to control how many threads are assigned to this.

 

So the CPU decompression already exists in DS1.1

 

So if a game is only going to be released for Windows, DS1.1 the backwards compatibility already exists. So there is no reason NOT to use it.

 

But GDeflate on other platforms? Well they say they will release the code under the Apache license. So implementing it on other GPU's shouldn't be a problem, even if it's implemented as OpenCL.

 

The Nvidia library appears to support an additional format developed at facebook https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md

 

 

[Quackers101]

They say one should enable an option to turn GPU decompression on/off

also can be cases where it decompresses on the CPU instead of the GPU.

"Rather than thinking in terms of compressing files, it is better to think about compressing streams."

Quote

in case they are running on a system that maybe has a significantly faster CPU than GPU.

Quote

- DirectStorage 1.1 doesn’t have a way to force CPU decompression on a per-request or per-queue basis. Instead, custom decompression can be used

 

- If the GPU/driver is unable to use the DirectCompute fallback, then DirectStorage will decompress on the CPU

- Recall that GDeflate splits each stream into tiles, where each tile decompresses to 64 KiB of data.

- Staging buffer sizes of around 128 MiB are required to fully saturate the IO stack.

 

Quote

GPU decompression is supported on all DirectX 12 + Shader Model 6.0 GPUs. However, one of the benefits of DirectStorage 1.1 is that GPU hardware vendors can provide additional optimizations for their hardware, called metacommands.

[Gamer Schnitzel]

When I built my PC in May 2021 I was considering a PCIE 4.0 SSD which would have costed more but I thought surely it is coming soon since the next gen consoles have it, right? I decided to skip it and save myself the money and am glad I have done so. It was all nothing more than marketing to boost SSD sales. Now I am going to build a new PC next month and have had absolutely zero need for a PCIE 4.0 SSD. Considering there still won't be a single game using DirectStorage next month I might just skip out on it again unless they're just cheap now.