Researchers use machine learning to make GTA V photorealistic

[gregg_goldstein]

Summary

Developers are overpromising gamers "photorealistic" graphics since the infamous days of the PS2's The Getaway. Setting aside what "photorealistic" means, we may be closer to that unicorn goal than ever before. Researchers at Intel Labs may soon have us covered by applying machine learning techniques to rendered footage captured from a console.

 

Quotes

Quote

We present an approach to enhancing the realism of synthetic images. The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines.

 

Stephan Richter --

Intel Labs

 

My thoughts

Before you jump in on relevant modding forums, they are turning a recording of a playthrough and applying a process which makes it look "photorealistic". Inference currently takes half a second. This is a step toward integrating real-time AI enhancement into rendering pipelines, but it’s not ready for an actual game. It’s not real time yet, and, it's not clear when we would have the GPU grunt available to apply any of the techniques discussed in the paper in real time. 

 

Sources

https://gizmodo.com/grand-theft-auto-looks-frighteningly-photorealistic-wit-1846878938

 

https://intel-isl.github.io/PhotorealismEnhancement/

[Helpful Tech Witch]

Cool. So like it's 2fps, but it's cool.

[igormp]

Seems to be a GAN-like architecture, neat.

[trag1c]

[BuckGup]

That's awesome. So within 10 years we will be playing VR games that are dynamically created in real time. Almost like the universe hmmmm

[manikyath]

to be completely honest here...

 

it's cool and all, but i watched trough their entire marketing blurb, and just about everything this thing does, seems to be only marginally better than just using higher res textures, more complex models, a more complex lighting engine, etc. in the game itself. essentially this cutting edge technology has been outdone by the way game modders have been improving visual fidelity for years.

 

essentially they took a game from 2013, and made it look like a game from 2021, using an amount of horsepower that we could see used 'realtime' in 2030.

 

maybe technology like this will be VERY good for actually producing high fidelity textures and models in future games, but i just dont see why this would need to ever be a thing for realtime rendering.

[Murasaki]

5 minutes ago, manikyath said:

to be completely honest here...

 

it's cool and all, but i watched trough their entire marketing blurb, and just about everything this thing does, seems to be only marginally better than just using higher res textures, more complex models, a more complex lighting engine, etc. in the game itself. essentially this cutting edge technology has been outdone by the way game modders have been improving visual fidelity for years.

 

essentially they took a game from 2013, and made it look like a game from 2021, using an amount of horsepower that we could see used 'realtime' in 2030.

 

maybe technology like this will be VERY good for actually producing high fidelity textures and models in future games, but i just dont see why this would need to ever be a thing for realtime rendering.

To me it looks like they applied the color grading of the footage to the game to produce some sort of filter. Looks like something that can be reproduced with some mod.

[manikyath]

Just now, Murasaki said:

To me it looks like they applied the color grading of the footage to the game to produce some sort of filter. Looks like something that can be reproduced with some mod.

the color shift comes from the fact they used a particularly "grey" image set for their reference, compared to the particularly "bright florida look" of GTA V.

 

it actually produces a lot of very nice detail, but for all the detail it creates, it just leaves a lot on the table because essentially the image processing pipeline still has no idea what the images are about.

The segment where they talk about the roads.. is literally outdone by the most basic texture replacing mod.

[manikyath]

Just now, gabrielcarvfer said:

It's nice and everything, but it uses engine data (depth, albedo, light map, etc). If it only used the video output it would be far more impressive.

the idea is that using engine data produces a much more accurate model than not using engine data. that's the whole point of the exercise as far as i gather from it.

[JZStudios]

11 minutes ago, Murasaki said:

To me it looks like they applied the color grading of the footage to the game to produce some sort of filter. Looks like something that can be reproduced with some mod.

There's some de-hazing of the mountain, along with changing specular values and textures at least. The color grading is seemingly because all of it's trained input was a crap camera video feed that looked like that so it's acting independently of actual lighting color information. So it's kind of a filter, but also kind of not, and it does actually tie elements together more than normal filters typically do.

[Bombastinator]

19 minutes ago, manikyath said:

to be completely honest here...

 

it's cool and all, but i watched trough their entire marketing blurb, and just about everything this thing does, seems to be only marginally better than just using higher res textures, more complex models, a more complex lighting engine, etc. in the game itself. essentially this cutting edge technology has been outdone by the way game modders have been improving visual fidelity for years.

 

essentially they took a game from 2013, and made it look like a game from 2021, using an amount of horsepower that we could see used 'realtime' in 2030.

 

maybe technology like this will be VERY good for actually producing high fidelity textures and models in future games, but i just dont see why this would need to ever be a thing for realtime rendering.

Tricks usually have to do with angle of view though.  They often work for only one point.  This would work for any point.  It would allow the creator to stop having to worry where the audience is and removed the requirement of staged performance.  I can’t conceptualize where this could go.  Might be nowhere I suppose but it could possibly be concept shifting 

[manikyath]

31 minutes ago, Bombastinator said:

Tricks usually have to do with angle of view though.  They often work for only one point.  This would work for any point.  It would allow the creator to stop having to worry where the audience is and removed the requirement of staged performance.  I can’t conceptualize where this could go.  Might be nowhere I suppose but it could possibly be concept shifting 

yes.. but in this topic, the option that is a "trick" is the machine learning algorithm. it only works because it has a dataset with images similar to where the player may be.

 

higher res textures work from any angle, complexer models work from any angle, a more complex lighting engine works from any angle.

And all of those combined are FAR easier to run than this complicated machine learning thing...

[Bombastinator]

3 minutes ago, manikyath said:

yes.. but in this topic, the option that is a "trick" is the machine learning algorithm. it only works because it has a dataset with images similar to where the player may be.

 

higher res textures work from any angle, complexer models work from any angle, a more complex lighting engine works from any angle.

And all of those combined are FAR easier to run than this complicated machine learning thing...

So you’re pointing out that there aren’t two types of rendering there are 3 (ray tracing, Rasterizing, and machine learning) and the machine learning system is less similar to how things work rather than more similar, and is doing its work by running from examples.  It would be potentially much more able limited then.  So the machine learning system Makes an interesting check, but that’s about it.  It does point to one factor though: Bay Area Light.  There was a painting school called the Bay Area School that was affected by the large amount of sunlight in the area where the painting took place.  The machine learning stuff shows more Bay Area light than the rendered stuff does.  It points out a hole. 

I may have conflated ray tracing and machine learning here.

[manikyath]

3 minutes ago, Bombastinator said:

So you’re pointing out that there aren’t two types of rendering there are 3 (ray tracing, Rasterizing, and machine learning) and the machine learning system is less similar to how things work rather than more similar, and is doing its work by running from examples.  It would be potentially much more able limited then.  So the machine learning system Makes an interesting check, but that’s about it.  It does point to one factor though: Bay Area Light.  There was a painting school called the Bay Area School that was affected by the large amount of sunlight in the area where the painting took place.  The machine learning stuff shows more Bay Area light than the rendered stuff does.  It points out a hole. 

I may have conflated ray tracing and machine learning here.

my point is that everything this machine learning approach does, can also be achieved within the game's engine, for less compute cost.

 

it's an interesting thought experiment, it probably has some use case outside of gaming, but in the scope of 'making GTA V look better', it is probably the most convoluted inefficient way to have a relatively acceptable result.

[Fasterthannothing]

1 hour ago, Murasaki said:

To me it looks like they applied the color grading of the footage to the game to produce some sort of filter. Looks like something that can be reproduced with some mod.

Thank goodness it's not just me. I was like is this supposed to look better? Looks like something from now with a filter on it.

[Bombastinator]

10 minutes ago, manikyath said:

my point is that everything this machine learning approach does, can also be achieved within the game's engine, for less compute cost.

 

it's an interesting thought experiment, it probably has some use case outside of gaming, but in the scope of 'making GTA V look better', it is probably the most convoluted inefficient way to have a relatively acceptable result.

Perhaps. The result is at least in some cases vastly higher quality though.  It DOES look better, at least in that particular example.  It may look worse elsewhere. In particular it’s sampled real life sun rays to show the difference. I suspect neither system is going to work well all the time.  As far as it being convoluted and inefficient, it clearly does take more processing power than rasterizing.  So “more” clearly.  “The” may be strong. The possibility of Rube Goldberging   something even less efficient int existence always exists.

[manikyath]

1 minute ago, Bombastinator said:

Perhaps. The result is at least in some cases vastly higher quality though.  It DOES look better, at least in that particular example.  It may look worse elsewhere. In particular it’s sampled real life sun rays to show the difference. I suspect neither system is going to work well all the time.  As far as it being convoluted and inefficient, it clearly does take more processing power than rasterizing.  So “more” clearly.  “The” may be strong. The possibility of Rube Goldberging   something even less efficient int existence always exists.

it looks better than the original game from 2013, it doesnt look better than loading the game up with visual mods and high-res textures.

and loading the original game with mods can still run realtime, the proposed machine learning algorithm cant.

[CerealExperimentsLain]

1 hour ago, manikyath said:

to be completely honest here...

 

it's cool and all, but i watched trough their entire marketing blurb, and just about everything this thing does, seems to be only marginally better than just using higher res textures, more complex models, a more complex lighting engine, etc. in the game itself. essentially this cutting edge technology has been outdone by the way game modders have been improving visual fidelity for years.

 

essentially they took a game from 2013, and made it look like a game from 2021, using an amount of horsepower that we could see used 'realtime' in 2030.

 

maybe technology like this will be VERY good for actually producing high fidelity textures and models in future games, but i just dont see why this would need to ever be a thing for realtime rendering.

Is it even that?  It seems to be mostly just color correction in a way that makes it look like a dash cam footage from a video on Youtube called 'LA DRIVER FLIPS HIS CAR LOL'.  It's just... Drab and flat and green.  Yes it looks 'real' but it looks like a boring kind of real, it kills some of the colorful magic that's a bit super-real in GTA5.

 

I imagine the AI is largely trained on dash cam footage, hence resulting it it looking like that, but I'm not sure that's an 'improvement' from an objective and artistic view.  Well, at least if your goal is doing anything other than making fake dash cam crash videos for YouTube.

[Bombastinator]

7 minutes ago, manikyath said:

it looks better than the original game from 2013, it doesnt look better than loading the game up with visual mods and high-res textures.

and loading the original game with mods can still run realtime, the proposed machine learning algorithm cant.

So you’re saying the comparison is not accurate because the rasterized sample is out of date and not actually a product of modern rasterization.  Effectively  An intel sales graph that has no actual useful information in it and is merely intended to mislead.

[manikyath]

Just now, Bombastinator said:

So you’re saying the comparison is not accurate because the rasterized sample is out of date and not actually y’all y’all a product of modern rasterization.  Effectively  An intel sales graph that has no actual useful information in it.

i'm saying that making GTA V footage more "photo-realistic" is REALLY low hanging fruit at this point.

it's like saying you made a shader to improve java minecraft's lighting engine, your starting point is so basic that as long as you dont cause hilareous visual glitches, your result will be better than what you started out with.

 

3 minutes ago, CerealExperimentsLain said:

Is it even that?  It seems to be mostly just color correction in a way that makes it look like a dash cam footage from a video on Youtube called 'LA DRIVER FLIPS HIS CAR LOL'.  It's just... Drab and flat and green.  Yes it looks 'real' but it looks like a boring kind of real, it kills some of the colorful magic that's a bit super-real in GTA5.

 

I imagine the AI is largely trained on dash cam footage, hence resulting it it looking like that, but I'm not sure that's an 'improvement' from an objective and artistic view.  Well, at least if your goal is doing anything other than making fake dash cam crash videos for YouTube.

go find their marketing blurb, they just have a particularly grey reference imageset. why they did that? no idea.. because that's what they had on hand, i presume?

[Bombastinator]

3 minutes ago, manikyath said:

i'm saying that making GTA V footage more "photo-realistic" is REALLY low hanging fruit at this point.

it's like saying you made a shader to improve java minecraft's lighting engine, your starting point is so basic that as long as you dont cause hilareous visual glitches, your result will be better than what you started out with.

 

go find their marketing blurb, they just have a particularly grey reference imageset. why they did that? no idea.. because that's what they had on hand, i presume?

Yes.  Intel has shown a habit lately of avoiding actual comparisons to competing products and instead show comparisons to earlier no longer current systems hoping that people will assume that the comparisons ARE to actual competing products.  Misleading. 

Edited May 13, 2021 by Bombastinator

[Kisai]

1 hour ago, manikyath said:

to be completely honest here...

 

it's cool and all, but i watched trough their entire marketing blurb, and just about everything this thing does, seems to be only marginally better than just using higher res textures, more complex models, a more complex lighting engine, etc. in the game itself. essentially this cutting edge technology has been outdone by the way game modders have been improving visual fidelity for years.

 

 

Modders are still limited by the original game engine, and a lot of bad-faith modding tends to be unreliable.

 

Like it doesn't require a lot of effort to "increase" the visual quality of a game by unlocking various tunable features that already exist in the game, and we've been told "photo-realistic" with every generation of game consoles since the ps2, but the reality is that we will never have photo-realistic games.

 

1) Raytracing + radiosity + caustics is required, which means games have to be created around being ray-traced, not merely bolted on (eg minecraft), this is immensely math intensive and hard to parallelize. So we won't see this except maybe with high-end VR HMD's, since that's where it's needed (each eye needs a different image.)

 

2) ML based "upscaling" and "quality increasing" tends to rely on the style-transfer mechanism that ML is fairly good at. But that doesn't mean it's photo-realistic. Just looking at the images themselves doesn't suggest any kind of photo-realism.

Like to me this doesn't really look any more "photo-realistic" than the game itself, there's some improvement to the lighting, but I strongly feel that the input photos were not actually 36-bit HDR. So what you are believing to be "photo-realistic" is really just style-transferring the camera quality, rolling-shutter and all.

https://intel-isl.github.io/PhotorealismEnhancement/

 

Like the only detail that seems to substantially change is the road texture, which looks like it was dialed back, while the grass color was dialed upwards. Like in many of the images I see warm colors being changed with cool ones. As if a layer of smog was removed.

 

 

 

[porina]

3 hours ago, gregg_goldstein said:

they are turning a recording of a playthrough and applying a process which makes it look "photorealistic"

My understanding was they use more than just the final output rendering in order to achieve that output, taking also other data from the game engine into consideration.

 

I saw the video on youtube earlier. My take on it is to see this like a kinda DLSS but turned up to over 9000. A technique doesn't have to be end user relevant on day 1. If it did, we'd never get anything done. Research like this starts the discussion, what works, what doesn't, and that can be built upon. For example, imagine a scenario where you might intentionally render lower quality and rely on the processing to give a nice output. So similar to DLSS but working at a lower level within the game engine. Alternatively imagine you could have very different visual styles in games based on the same lower level rendering. This might be used by the game itself as an easier way to generate different feeling areas without having to create new assets explicitly each time.

[Delicieuxz]

'Check out our video of photorealistic graphics - but only in 144p to 720p'

 

They coulda made a bit more effort to show-off their tech, with a higher-res video.

[Bombastinator]

29 minutes ago, porina said:

My understanding was they use more than just the final output rendering in order to achieve that output, taking also other data from the game engine into consideration.

 

I saw the video on youtube earlier. My take on it is to see this like a kinda DLSS but turned up to over 9000. A technique doesn't have to be end user relevant on day 1. If it did, we'd never get anything done. Research like this starts the discussion, what works, what doesn't, and that can be built upon. For example, imagine a scenario where you might intentionally render lower quality and rely on the processing to give a nice output. So similar to DLSS but working at a lower level within the game engine. Alternatively imagine you could have very different visual styles in games based on the same lower level rendering. This might be used by the game itself as an easier way to generate different feeling areas without having to create new assets explicitly each time.

Makes me wonder how different this is from AMD’s DLSS competitor