dlss I Wonder When AI Upscaling Will Reach YouTube and Other Video Streaming Services

[Mark Kaine]

definetly see the benefits of this, personally I hate upscaling though, on the other hand bitrates/ compression on yt is so awful it just might help (only videos that look actually good for me on yt are "4k" even though I only have a 1080p monitor... still get the benefit of not trash tier bitrates obviously) 

[igormp]

The problem is the sheer lack of proper HW in devices to do so. People with RTX GPUs would run that without problems, and so would iPhone users (since the newest ones have tons of ML coprocessors). The rest of the population wouldn't benefit that much since they barely have the latest hw decoders available.

[BlueChinchillaEatingDorito]

Does that mean Smash Mouth will finally look good? 

 

[LAwLz]

22 hours ago, Briggsy said:

Youtubers uploading in 4K could instead upload in 1080p and an AI upscaler could upscale to 4K and beyond. For Google's benefit it would be better served if the upscaling were done on the client side, so that Google only has to send 1080p or even less across the web. 

I'm sure Google would love to be able to serve a 1080p video and have it upscaled to 4K on the client. But the problem is that upscaling that way requires a lot of processing power on the end device, or specialized hardware.

So like you said in your post, it would have to be done on the client side. If it's done on the client side, and requires specialized hardware in the client's device, then there isn't much Google can do about the situation. 

 

 

22 hours ago, RejZoR said:

As much as NVIDIA is hyping DLSS to be "Ai powered upscaling", you simply can't create things that are missing. It's just impossible no matter how much "Ai" buzzwords you throw at it. And DLSS is not creating detail out of nothing, they are just highly selectively processing the rendering to maximize performance while not affecting quality on a perceivable level. Imagine placing a 100% JPEG quality downgrade on a lossless PNG photo. There is no way anyone can pinpoint the quality downgrade, but the image will be 50% smaller than original. That's DLSS. No one but NVIDIA knows the exact specifics, but you can be sure it's not just magically making all the details out of nothing. That's just impossible, especially in highly dynamic worlds like in games that are fully interactive and don't have a predictable angle or movement of anything in it.

 

And that further applies to video. Unlike games where each individual frame is a stand alone sharp rendered frame of a rasterized scene, frames in movies are image captures of something that has happened. If it's smudged, it's smudged on capture. In games, they fake it with motion blur effect on purpose to give that effect. On top of a still sharp rasterized frame. They have pretty much free hands to manipulate every frame in games though through the rendering pipeline. You can even render it at 320x480 or 3840x2160 and it will retain all the bits, you'll just see more of them at 4K. With video, if it was captured at 320x480, that's it. You can make up some details using whatever Ai interpolation, but you can't possibly add back the details that weren't captured. Reason they could remaster some of old movies that were still recorded on actual tape is because with video on tapes, you actually have the capability to extract far higher resolution imagery from the physical tape than we were able to display on TV's or project in cinemas. With digital recording today, what you capture at 8K is 8K and that's it. You'll never be able to up that to 16K in the future like we did with taped movies from the past. It's just physically impossible.

It's entirely possible to create things that are missing.

We already have several programs and functions that does this, and does it well too. DLSS is one of them.

As long as you don't worry about recreating everything 1:1 to some original, and instead focus on creating details that will look like they belong there to a human, then it's good enough. 

[RejZoR]

Thats not how it works. It's not how any of it works. Sticking a face on a prerecorded video in a lenghty processing is not the same as doing it in live 3D game where nothing is prerecorded and has to be done in realtime.

[igormp]

On 9/27/2020 at 12:16 PM, RejZoR said:

Press X for doubt...

Bumping this just to bring something really cool: we're now able to do things such as reverse-crops in videos! 

 

[LAwLz]

On 9/28/2020 at 10:07 AM, RejZoR said:

Thats not how it works. It's not how any of it works. Sticking a face on a prerecorded video in a lenghty processing is not the same as doing it in live 3D game where nothing is prerecorded and has to be done in realtime.

No, you're the one who doesn't understand how any of this works.

We already have working products using it. We already know how well it works. If you don't believe reviews of Nvidia's video upscaling capabilities then there are several papers about high quality image upscaling techniques available for you to read.

 

Who said anything about "sticking a face on a prerecorded video"? That's not what these upscaling techniques does.

This is a picture of the US flag. Let's imagine I had trained an AI to know what the US flag looks like by showing it high resolution images. 

Spoiler

 

If I fed the AI a really blurry and badly upscaled image of the flag, do you not think that an AI could be sophisticated enough to clean it up?

Spoiler

 

 

[NineEyeRon]

https://imgflip.com/i/4ikii5

[sb2702]

If anyone is interested in trying out AI Upscaling on YouTube, I built a Chrome Extension which does that.

 

https://chrome.google.com/webstore/detail/ai-video-upscaler-alpha/noegpndjakocogidclhgkocfmhjkjgjj/

 

AI Upscaling takes some compute power, so we've restricted it to 240p and 360p upscaling. I've seen newer devices handle much more, and I think that newer AI-specific hardware and Associated APIs (https://www.w3.org/2020/Talks/mlws/nh_webnn.pdf) will make it more feasible to do 1080p -> 4K upscaling for YouTube videos on average laptops/tablets.

 

Till then, you need a GPU or a smart TV.

 

 

 

[WoodenToaster]

On 9/27/2020 at 10:45 AM, minibois said:

This is one thing they won't be able to do right now for a couple reasons.

 

1. Bitrate: partially why 4K content looks good on YouTube is the upper vitrage vs. 1080p content. When serving 1080p content, they need to upscale and also remove artifacts from low bitrate

 

2. Compute vs. storage: right now YouTube needs to save a 4k, 1080p, etc. Version of a file that takes up a lot of storage.

What you imply should happen is lower the amount of storage needed, but up the amount of compute power needed.

That's a trade off, which YouTube need to make a decision on , but it think at the moment they will stick with the storage needed.

It wont be impossible anymore here in 2022, Hey, its me from the future. now since you might of neglected a small detail. Since YouTube's low bit Bitrate is server side, in short they have the full HD vid on hand. Its just sending it out as a lower rate. That means if Upscaling is applied with a data file including trained prerenders of the video beforehand, it could be sent via small package that can be used to recreate the image as if it had Higher bit rate client-side.

 

It wont be DLSS but it would be technically Upscaling the image trout AI. YouTube needs less bandwidth to send you your video saving money and you get a better image in theory. Sending that package could be a little as 25MB* since its only trained information on that video. (Size my very from the amount of compression and length of the video)

 

The only down side is, that they need to actually take computing power to make that package for every video (Takes hours on our hardware for just a 20min 4K Vid). We use a sort of thing just like that in my line of work. A 4K, low Bit vid gets upscaled trough a AI interpreter and gets sent with both files, My pc takes those files and uses a Software that reads those files and sends it to the AI interpreter that uses those data to up the bitrate almost as if it was never tampered with, Almost. Small artefacts will happen from time to time since file is trained on the Original video and compressed. Decompression might lead to some unwanted results, but as far as I know it works great. I just don't thing YouTube will run extra Hardware just to do that, even tough the tech is there to be taken. But that does lead to the same concussion you made. YouTube wont be ready to invest in that, I cant imagen the costs of having to even kickstart a dedicated YouTube Upscaling Interpreter App and having to use dedicated Hardware to Train each video and Compress the results just to send it out. Hope this was interesting non the less

 

(I know nothing, barley anything of this tech to be honest. I just Emailed a Co-worker that knows this stuff and asked how all this works. I technically copy and pasted most of what she said, Smarter every day I say)