More info on AMD's Super Resolution: machine learning based?

[igormp]

Summary

A patent from 2019 by AMD detailing a little bit about their DLSS competitor was made public yesterday, and it does seem to use machine learning, just like DLSS does.

 

Quotes

Quote

[…]  A super resolution processing method is provided which improves processing performance. The method includes receiving an input image having a first resolution, generating linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generating non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The method also includes converting the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and providing the output image for display. […]

 

 

My thoughts

Since that's somewhat old, we can't be sure that's the solution they'll be using in their final product. Nevertheless, it gives us an insight on what they were working, which is deep learning-based, just like DLSS.

I wonder how would that idea fare on a GPU without Tensor cores/Matrix cores (only available on instinct GPUs), and how bad it would be on a CPU. There's also no mention of how they'd deal with the temporal issues of supersampling, showing only the solution required for a single frame.

 

Sources

Patent

Images

Videocardz

[CephDigital]

Huh. Me taking AI modules for uni is being useful! I can somewhat understand that patent and how it works.

[Humbug]

I suspect they are going to continuously improve their implementation over time. So even if the launch version does not have machine learning AI as detailed in this patent, I bet it would come down the line as a way of improving the image quality further....

[GDRRiley]

I've heard some rumors say its already better than DLSS 1.0 or lower resolutions with AA (which it better be) as they are low bars
but also that its a lot easier to add in

[Humbug]

 

7 minutes ago, GDRRiley said:

I've heard some rumors say its already better than DLSS 1.0 or lower resolutions with AA (which it better be) as they are low bars
but also that its a lot easier to add in

Ya. The rumors I heard are that the image quality is better than DLSS 1.0... but that DLSS 2.0 quality mode still has an edge

[igormp]

2 hours ago, Humbug said:

I suspect they are going to continuously improve their implementation over time. So even if the launch version does not have machine learning AI as detailed in this patent, I bet it would come down the line as a way of improving the image quality further....

The problem is that they're really late to the game, so everyone expects something good given the time they had, which causes them to take even more time to refine what they have, and so on.

Even though DLSS 1.0 was kinda shit, it was the only thing available and had no competitors, so it could evolve freely only getting better against its previous versions, while AMD has to play catch up with what nvidia has, which gets even harder considering that their sw department is really lacking, and they have almost no R&D geared towards ML.

[LAwLz]

3 hours ago, GDRRiley said:

I've heard some rumors say its already better than DLSS 1.0 or lower resolutions with AA (which it better be) as they are low bars
but also that its a lot easier to add in

Don't trust rumors. Especially not one a# empty and unverifiable as that. Wait and judge it with your own eyes (or better yet, an objective measurement). 

[porina]

8 hours ago, igormp said:

Since that's somewhat old, we can't be sure that's the solution they'll be using in their final product.

Do not assume that because a patent exists it will ever be used. Many companies will patent anything that is patentable as it might be worth something to someone else.

 

1 hour ago, LAwLz said:

Wait and judge it with your own eyes (or better yet, an objective measurement). 

Stick with using eyes for this type of comparison. What looks good or not is more complex than what objective measurements can reliably provide. It will likely be the case that people wont even agree on what looks best.

 

Earlier today I watched part of a video comparing Metro Exodus with it's recent Enhanced Edition, and the differences and tradeoffs between native rendering, DLSS 1, and DLSS 2 were interesting to say the least.

[LAwLz]

9 minutes ago, porina said:

Stick with using eyes for this type of comparison. What looks good or not is more complex than what objective measurements can reliably provide. It will likely be the case that people wont even agree on what looks best.

Maybe in a blind test. I don't trust that people can be objective when doing these comparisons. I am fairly sure that in non-blindtests I already know which people will say AMD's implementation is superior to Nvidia regardless of how the image actually looks. 

[IkeaGnome]

10 minutes ago, porina said:

Stick with using eyes for this type of comparison. What looks good or not is more complex than what objective measurements can reliably provide. It will likely be the case that people wont even agree on what looks best.

 

Earlier today I watched part of a video comparing Metro Exodus with it's recent Enhanced Edition, and the differences and tradeoffs between native rendering, DLSS 1, and DLSS 2 were interesting to say the least.

I'd be curious to see this as a "blind" test like they recently did with DLSS. 

[igormp]

1 hour ago, porina said:

Stick with using eyes for this type of comparison. What looks good or not is more complex than what objective measurements can reliably provide. It will likely be the case that people wont even agree on what looks best.

There are some objective measurements that are really good though. My favorite is VMAF, but other classical ones such as PSNR, SSIM and STRRED are good, specially when used together.

[porina]

7 hours ago, igormp said:

There are some objective measurements that are really good though. My favorite is VMAF, but other classical ones such as PSNR, SSIM and STRRED are good, specially when used together.

I made my comment based on experience in the audio domain, where similarly there were various tools promising to give you a MOS value that correlates with real human testing. Generally they weren't that bad, but you often had strict constraints on the conditions involved. 

 

I'm not familiar with the video side but it is not a stretch that similar problems would exist. Skimming through the VMAF link they say it works per-frame, and they use averaging to gain some temporal information. One of the potential artefacts with upscaling is (lack of) temporal stability, which can cause annoying flickering. That model will be blind to that.

[tim0901]

On 5/21/2021 at 3:08 PM, igormp said:

I wonder how would that idea fare on a GPU without Tensor cores/Matrix cores (only available on instinct GPUs), and how bad it would be on a CPU.

This won't be done on the CPU, simply because of the inefficiency of doing so. Sending that much data from the GPU framebuffer (~53MB per frame at 4k) to the CPU and back will take a not-insignificant amount of time. And this problem only gets worse and worse the higher your framerate. At 100fps, this transfer alone would be using a third of the bandwidth provided by an x16 PCI-E 4.0 link, leaving much less time per frame for any processing at either end and likely restricting the feature to systems with that full 4.0x16 link available. (That is unless you were to introduce a frame delay, but that would be rather unpopular in fast paced titles...)

 

Instead, it will almost certainly be done on the general compute cores of the GPU (the "stream processors") thereby saving time and eliminating any out-of-card problems. But this has the drawback of pulling those resources away from other parts of the rendering process, because "serving" a trained AI model isn't free. It still requires a not-insignificant amount of resources and, as before, this demand will scale with the framerate. DLSS doesn't have this problem - all this work is offloaded to the tensor cores (of which there are plenty), meaning the technique doesn't have an impact on the rest of the rendering pipeline.

 

What this all means is that AMD's job is harder than Nvidia's. Lets say in a hypothetical scenario that both the 3090 and 6800xt get an identical 100fps in a title normally. If DLSS provides a theoretical 25% uplift, then the 3090 runs it at 125fps. However if AMD SR also provides a 25% theoretical uplift, but at the cost of using 3% of the GPU, then the 6800xt would be sitting at ~121fps - a real-world uplift of 21%.

 

Which means that if AMD SR is to try and compete with DLSS, then it needs to be better than DLSS in order to be equal with it. As such my expectations for it are... limited. I want them to succeed, but given AMD's nonexistant history with AI I'm not expecting miracles. As such, I'm worried that AMD will sacrifice image quality for fps (for the benchmarks) which - as DLSS 1.0 showed - people aren't going to be happy with, further giving the technology a bad rep and therefore reducing its adoption rate by game devs.

[igormp]

12 hours ago, tim0901 said:

This won't be done on the CPU, simply because of the inefficiency of doing so. Sending that much data from the GPU framebuffer (~53MB per frame at 4k) to the CPU and back will take a not-insignificant amount of time. And this problem only gets worse and worse the higher your framerate. At 100fps, this transfer alone would be using a third of the bandwidth provided by an x16 PCI-E 4.0 link, leaving much less time per frame for any processing at either end and likely restricting the feature to systems with that full 4.0x16 link available. (That is unless you were to introduce a frame delay, but that would be rather unpopular in fast paced titles...)

I wasn't the one who mentioned it'll be done on a CPU, the patent itself did so. And you can achieve zero-copy inference when you have integrated GPUs that share RAM with the CPU itself, so the overhead between transfers is also a non-issue.

You're also assuming this is only meant for games, whereas one could also use it to supersample videos and other kinds of media (as seen on the nvidia shield). For that case, it should be doable on a CPU as long as their inference times are good enough, and, as you said, it'll probably sacrifice image quality in order to achieve good enough frame rates.

 

Other than that, I agree with what you said. AMD has a poor track record when it comes do anything ML, their consumer platforms lack any dedicated hardware for matrix FMA (only available in their instinct lineup, with no proper software support either), and saying that their solution will be platform agnostic makes me really wonder how well it'll work. I wouldn't doubt it they were relying on DirectML to achieve their results, but then you'll be locked onto Windows-based platforms.

[RejZoR]

People are willing to give AMD a pass for 1 time. They've done so for ray tracing on RX 6000 cards and I think they are willing to forgive it for this feature too. They kinda did the same for DLSS too. It did the promising, but poorly. DLSS 2.0 did deliver and people are happy. Same is expected from AMD now. If they can pull it off first time, great, but I have doubts they will.

[Fnige]

On 5/22/2021 at 1:00 AM, CephDigital said:

Huh. Me taking AI modules for uni is being useful! I can somewhat understand that patent and how it works.

are you able to dumb it down pls for people like me

[tim0901]

19 hours ago, igormp said:

I wasn't the one who mentioned it'll be done on a CPU, the patent itself did so. And you can achieve zero-copy inference when you have integrated GPUs that share RAM with the CPU itself, so the overhead between transfers is also a non-issue.

 

Indeed I hadn't considered iGPUs. Certainly there it would make more sense to run on the CPU - especially on consoles where you have a fixed set of hardware.

 

19 hours ago, igormp said:

You're also assuming this is only meant for games, whereas one could also use it to supersample videos and other kinds of media (as seen on the nvidia shield). For that case, it should be doable on a CPU as long as their inference times are good enough, and, as you said, it'll probably sacrifice image quality in order to achieve good enough frame rates.

Yes, because DLSS can't do this. Even DLSS 2.0 requires information provided by the game outside of the video signal - this is why it can't be enabled as a driver-level feature that works on every game out there. The Nvidia shield doesn't use DLSS to upscale video - it uses its own AI model to do this - which adds up with the marketing for the feature not mentioning DLSS anywhere.

 

As far as AMD's marketing is concerned, we've seen no reason to suggest that video upscaling is a feature enabled by AMD SR either - everything they've mentioned has been purely centred on gaming. So yes, I am assuming this is for gaming, as we've no evidence to suggest otherwise. The patent just uses the phrase "video stream" - never getting more specific than that - so suggesting that the final product will be able to upscale video as well as gaming signals is pure speculation at this time. It is, after all, just a patent, meaning there's no guarantee that anything written on it will ever come to market.

 

That being said, I can't see any reason why AMD couldn't create a solution to match the upscaling found in the Nvidia Shield - the limiting factor would be their own AI capabilities. The Shield doesn't have tensor cores - even the 2019 models are using a Maxwell-based GPU - so any upscaling there will be performed using CUDA.

[CephDigital]

17 hours ago, SlimyPython said:

are you able to dumb it down pls for people like me

Unfortunately not, Im no good at explaining things xD

[Vishera]

On 5/21/2021 at 2:08 PM, igormp said:

Matrix cores

Matrix cores?

Wow,that name

Lisa Su should wear an outfit from the movie with the sunglasses and everything at the launch event.