Google Stadia to use AI to predict gamer's actions

[WkdPaul]

So apparently, Google is going to use AI to predict a user's action so that it will have "negative lag" ... not sure I understand how that would work. If you're not doing the action the AI predicted, will the game not work? I mean, if the AI thinks you're going to go left and does that, but you turn right, or start firing your weapon, won't the AI have already made you go left and that would exacerbates the lag?

 

Quote

“Ultimately, we think in a year or two we'll have games that are running faster and feel more responsive in the cloud than they do locally, regardless of how powerful the local machine is.”

 

[...]

 

Google believes Stadia can beat the PS5, Xbox Project Scarlett and any gaming PC or gaming laptop by using artificial intelligence. One of the tricks that Bakar described is “negative latency”. 

 

[...]

 

To avoid this and enable 4K and 8K resolutions without any lag, Google wants to use AI that predicts what a user will click next. That way, a server in the cloud will basically be able to pre-render ahead of your actual action, delivering these frames in advance so, when you actually click, you will get the right frame. 

If you click another button, then you will of course take whatever path you have chosen. But, statistically, this method will accelerate gaming and provide this negative latency in which everything feels immediate because it has already been pre-calculated before you made the decision.

 

Source ;

https://www.tomsguide.com/news/google-claims-stadia-will-outperform-consoles-by-predicting-players-moves

https://www.engadget.com/2019/10/10/google-stadia-negative-latency

[Origami Cactus]

I kinda understand that?

So it will predict what you would want to do, lets say go right. Then it already starts rendering the frames as if you went right. 

Now, you press the key and stadia receives the keypress. If it was right, it can already send over the rendered frames, but if you actually pressed left, it starts rendering the game as if you went left, and that action will have the "normal" amount of lag.

[RejZoR]

Bullshit. You can't predict something entirely unpredictable. This isn't a movie where you have a set of data and you can plan ahead how to apply filters for upcoming frames. Games are never the same twice, not even the most repetitive ones. When player actually does something, then you capture that and stream content to the player. No bullshit Ai buzzword can solve that. Ever. Stating otherwise is like saying you can bypass laws of thermodynamics. It just can't be done. What can be done is having stupid high bandwidth and stupid low latency. And stupid fast encoding that can render, capture and stream said video at negligible latencies on top of latencies from connection itself. So, yeah, trying to defy laws of thermodynamics basically...

[The King of the Undead]

to quote doc hudson "if you go hard enough left you'll go right." Because the AI decided to die.

[Origami Cactus]

2 minutes ago, RejZoR said:

Bullshit. You can't predict something entirely unpredictable. This isn't a movie where you have a set of data and you can plan ahead how to apply filters for upcoming frames. Games are never the same twice, not even the most repetitive ones. When player actually does something, then you capture that and stream content to the player. No bullshit Ai buzzword can solve that. Ever. Stating otherwise is like saying you can bypass laws of thermodynamics. It just can't be done. What can be done is having stupid high bandwidth and stupid low latency. And stupid fast encoding that can render, capture and stream said video at negligible latencies on top of latencies from connection itself. So, yeah, trying to defy laws of thermodynamics basically...

Yeah, while i can somewhat understand the concept, i can't see at all how they would apply it to a 3D video game, there is just so much movement (3axis for movement, 3x for camera, and all the other actions), they would have to render like a 100000 instances of the game at once to have a somewhat accurate prediction at hand. 

[Taf the Ghost]

So it'll do nothing until it feels like you're controller has gone unresponsive, then it'll snap back. If you're playing something like a platformer, after a couple of failed jumps, you're just going to give up on the game.

 

Google is basically using Ghosting as a selling point, lol.

[The King of the Undead]

Just now, Origami Cactus said:

Yeah, while i can somewhat understand the concept, i can't see at all how they would apply it to a 3D video game, there is just so much movement (3axis for movement, 3x for camera, and all the other actions), they would have to render like a 100000 instances of the game at once to have a somewhat accurate prediction at hand. 

about 2 epyc chips for that.

[WkdPaul]

5 minutes ago, Origami Cactus said:

I kinda understand that?

So it will predict what you would want to do, lets say go right. Then it already starts rendering the frames as if you went right. 

Now, you press the key and stadia receives the keypress. If it was right, it can already send over the rendered frames, but if you actually pressed left, it starts rendering the game as if you went left, and that action will have the "normal" amount of lag.

Yeah, but to be able to do that, it means it'll have multiple instance of the game running? Like 1 for the AI prediction and 1 if it gets it wrong? I mean, I totally understand AI prediction, I'm just at a loss about how they're going to be able to implement that.

[Taf the Ghost]

Just now, wkdpaul said:

Yeah, but to be able to do that, it means it'll have multiple instance of the game running? Like 1 for the AI prediction and 1 if it gets it wrong? I mean, I totally understand AI prediction, I'm just at a loss about how they're going to be able to implement that.

They'll do some sort of predict input system that'll come across as input stutter. It'll be like playing a badly coded Win95 game again because the keyboard & mouse polling rates were much lower.

[WereCatf]

I can only see this working for games without analog or analog-like input, ie. analog sticks in gamepads, a mouse or such, because the input from those can vary way too wildly from moment to moment, even for a single person. Basically, it wouldn't work for almost anything and be just a waste of power.

[WkdPaul]

also, "negative lag / latency" ... pretty click-bait IMO, and I refuse to use that expression! 

[jiyeon]

So how do Google expect to predict what I do with how eratic my gaming behaviours are?

 

That's like me playing Counter-Strike: Global Offensive and the AI 'predicting' me going lower tunnels in dust_2 and rendering the frames there early when actually I've gone to A because I've got an AWP rather than a rifle.

[WereCatf]

6 minutes ago, sowon said:

So how do Google expect to predict what I do with how eratic my gaming behaviours are?

 

That's like me playing Counter-Strike: Global Offensive and the AI 'predicting' me going lower tunnels in dust_2 and rendering the frames there early when actually I've gone to A because I've got an AWP rather than a rifle.

That's not actually that difficult. People tend to take certain routes more often than others and it's easy enough to log a bunch of variables and project estimated actions. The part I was talking above is about input-range, which is the actually hard part: when you e.g. push on an analog stick to the left, sometimes you push it only to 35 degrees, sometimes to 29 degrees, sometimes you have an unexpected twitch and push it all the way to 44 degrees, not to mention the speed at which you do this varies literally every single time and is never linear movement -- it tends to be more of a slow-rise, then a bell, then a slow-fall -- which all combine to make it very difficult to actually do it for anything with free-form movement and/or look.

[WereCatf]

In relation to this story, Google did mention some time ago one thing they do with the Stadia that should shave a few milliseconds off the latency: the controller itself is actually connected to the servers, instead of the signals from the controller first going to the console and then from the console to the servers -- it skips a couple of steps in-between compared to traditional controller-schemes. That's a perfectly reasonable and valid way of reducing some latency.

[Brooksie359]

I call bullshit on this one. Google says it will reduce lag compared to a modern day computer and be faster? Even at 60 fps it takes about 16ms to render a frame so they save 16 ms at 60fps which unless you have less than 16 ms for ping it will be slower. Also this is not bringing into play people with a good gaming pc capable of 144fps  in which case it goes down to 7ms. At 240 fps it goes down to about 4ms. I'm sorry but google is full of it. 

[RejZoR]

2 minutes ago, WereCatf said:

That's not actually that difficult. People tend to take certain routes more often than others and it's easy enough to log a bunch of variables and project estimated actions. The part I was talking above is about input-range, which is the actually hard part: when you e.g. push on an analog stick to the left, sometimes you push it only to 35 degrees, sometimes to 29 degrees, sometimes you have an unexpected twitch and push it all the way to 44 degrees, not to mention the speed at which you do this varies literally every single time and is never linear movement -- it tends to be more of a slow-rise, then a bell, then a slow-fall -- which all combine to make it very difficult to actually do it for anything with free-form movement and/or look.

Do you know how many angles mouse movement has when moving across X and Y axis? Now add Z axis for forward/backward motion. Now extrapolate that to aiming motion in a FPS game. Now multiply that with number of pixels you need to change on EVERY of those changes at 4K, 60 times a second if it'll run at 60fps. People don't seem to grasp it's not about "do I go to A or B site in CS:GO". It's about the absolutely exact path, aim and all the actions along the way I'm going to take to exact pixel. And you just CAN'T predict that. If you can, you've basically invented psychic superpower of predicting time itself. Aka, it just can't be done. Real life isn't Paycheck movie...

[WereCatf]

1 minute ago, RejZoR said:

Do you know how many angles mouse movement has when moving across X and Y axis? Now add Z axis for forward/backward motion. Now extrapolate that to aiming motion in a FPS game. Now multiply that with number of pixels you need to change on EVERY of those changes at 4K, 60 times a second if it'll run at 60fps. People don't seem to grasp it's not about "do I go to A or B site in CS:GO". It's about the absolutely exact path, aim and all the actions along the way I'm going to take to exact pixel. And you just CAN'T predict that. If you can, you've basically invented psychic superpower of predicting time itself. Aka, it just can't be done. Real life isn't Paycheck movie...

Um, you're just saying what I just said in different words.

[TechyBen]

30 minutes ago, RejZoR said:

Bullshit. You can't predict something entirely unpredictable. This isn't a movie where you have a set of data and you can plan ahead how to apply filters for upcoming frames. Games are never the same twice, not even the most repetitive ones. When player actually does something, then you capture that and stream content to the player. No bullshit Ai buzzword can solve that. Ever. Stating otherwise is like saying you can bypass laws of thermodynamics. It just can't be done. What can be done is having stupid high bandwidth and stupid low latency. And stupid fast encoding that can render, capture and stream said video at negligible latencies on top of latencies from connection itself. So, yeah, trying to defy laws of thermodynamics basically...

Lookahead on the emulators appears to actually render/code/run 4 or more "versions" where each is going up/down/left/right and then streams you the one closest to your actions... but this only removes/negates the native coded lage from the old games... not sure if it could "fix" existing lag.

[BuckGup]

I sorta doubt the extent at to which this works. If they can predict a gamers future actions then what's the difference than predicting a persons in the real world. If this is true than we are screwed harder than most thought as Google can now predict the future

[Curious Pineapple]

If you render at 4K with an increased FoV and only send the user the centre of the frame, you can effectively pre-render by changing which part of the frame the user sees and then use that as the centre point for the next frame.

[WereCatf]

8 minutes ago, Curious Pineapple said:

If you render at 4K with an increased FoV and only send the user the centre of the frame, you can effectively pre-render by changing which part of the frame the user sees and then use that as the centre point for the next frame.

Wouldn't work for 3D, it'd skew perspective.

[samcool55]

Won't this cause a lot of wasted compute power?

Sounds like a massive inefficient way to battle lag tbh.

[RejZoR]

3 minutes ago, Curious Pineapple said:

If you render at 4K with an increased FoV and only send the user the centre of the frame, you can effectively pre-render by changing which part of the frame the user sees and then use that as the centre point for the next frame.

Lol, it's all BS that CAN'T be done. If NVIDIA could pull something like this, they could render insane 3D scenes if they could literally predict time. But they can't do it even locally on their GPU where they have nanosecond latencies. And somehow ppl think Google can do it remotely through bunch of latency hoops measured in milliseconds. All I can say is LMAO. It can't be done. You can't interpolate non existing data and if you try you'll get horrendous artefacting on the output side. You can't "smooth out" millions of pixels of which existence you cannot predict in any way, shape or form.

 

There are tiling optimizations in remote software where only sections that change are sent over. But that works if you have a static screen and something small changes on it, like button state or text in a window. In 3D games, 99% of pixels on screen change at least 60 times a second. The other 1% is static GUI, assuming it's not even transparent at any degree. Which in 100% cases, it is. It's literally "player input, render action, capture it, stream it back to user to reflect his action". There is NO other way going around this. Locally rendered games only send actions data to server and to other clients. Here, they are sending entire video data. Something graphic cards require hundreds of gigabytes of throughput to materialize. So, yeah, lol.

[Franck]

This is going nowhere, if it would be a thing about generating optional frame that need to be coded for every single game. Most frames generated use a time counter of some sort to generate those frame at proper time. This is a complete rewrite of modules like physic engine, sound engine, etc. They are obviously simply talking about an AI version of what we called in the domain anti-lag feature.

[Kisai]

2 hours ago, WereCatf said:

In relation to this story, Google did mention some time ago one thing they do with the Stadia that should shave a few milliseconds off the latency: the controller itself is actually connected to the servers, instead of the signals from the controller first going to the console and then from the console to the servers -- it skips a couple of steps in-between compared to traditional controller-schemes. That's a perfectly reasonable and valid way of reducing some latency.

 

That kinda makes sense, but we're talking about maybe 4ms in the grand scheme of things assuming the user is on fiber and not dsl/cable/3g-umts/4g-lte.

 

In order of latency:

1. Distance from the data center. The only people who will have a good experience will be those who play only on their home realm (say SFO, NYC, MTL, or some other place with reasonably good connectivity and de-facto fiber availability to the equipment inside the home/unit.) You may still be looking at 6ms even if you're playing from a building next door to the data center.

2. Device hops inside your home. Some people have WiFi, some have wired connections, most of the time they're directly connected to the termination point inside the home, but then you have people who use their own routers, or have used Ethernet-over-coax/phone/power, wifi-extenders, and so forth that add latency twice.

3. Device hops between your home and your ISP backhaul. Some ISP's will send all their traffic to a central point in the city, or a neighboring city before sending it to another backhaul.

4. Lack of IPv6 support by ISP's, IPv4 carrier NAT, and so forth add routing latency

 

Quote

Tracing route to SITE_IN_SFO_AREA [2001:XXX:XXX:XXX::XXX]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  node-XXXX.ipv6.telus.net (Telus Gateway)
  2     8 ms     5 ms     7 ms  node-XXXXX.ipv6.telus.net 
  3     8 ms     9 ms     8 ms  XXXXX.bb.telus.com
  4     *        *        *     Request timed out.
  5    27 ms    27 ms    27 ms  XXXXXX (San Jose)
  6  2557 ms   993 ms   108 ms  XXXXXXX (Fremont) 
  7    26 ms    26 ms    26 ms  XXXXXXXXXX

So for example, IPv6, from Vancouver to a server in a Data center in the SF Bay area, the latency goes to 5ms just to get to Telus's equipment, 8ms at the carrier hotel, and is then 27ms as soon as it's in the SF Bay area. 27ms = 1.5frames of latency

 

IPv4 to the exact same machine:

Quote

Tracing route to keenspot.com [66.220.2.19]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  (Telus Gateway)
  2     7 ms    12 ms    13 ms  (Telus)
  3     8 ms     7 ms     7 ms  (Telus)
  4     8 ms     8 ms     8 ms  XXXXX.bb.telus.com 
  5    11 ms    11 ms    11 ms  XXXXXX (Portland)
  6    26 ms    26 ms    26 ms  XXXXXXX (Palo Alto)
  7    56 ms    29 ms    57 ms  XXXXXXXX (Fremont)
  8    27 ms    28 ms    27 ms  XXXXXXXXXX

One additional Telus hop, different route. Note that the Fremont node in both traces is pretty laggy. They're different ports on the same core routers as far as I know. Also note that the Telus gateway has an actual IPv6 address, where as the ipv4 address is the class c 192.168.x.x address.

 

So... just how much latency is there from here to Google? (trace 8.8.8.8)

Quote

 7     8 ms     8 ms     8 ms  dns.google [8.8.8.8]

8ms, ok so that's half a frame. But WHERE is that machine? It's very likely routed to an edge machine that's in Vancouver or Seattle. So let's try a more obvious address. Google.com

Quote

Tracing route to google.com [2607:f8b0:400a:808::200e]
over a maximum of 30 hops:

 

7     8 ms     8 ms     8 ms  sea15s11-in-x0e.1e100.net [2607:f8b0:400a:808::200e]

 

Tracing route to google.com [172.217.3.174]
over a maximum of 30 hops:

 

 7     8 ms     7 ms     7 ms  sea15s11-in-f14.1e100.net [172.217.3.174]

Not bad, but if this was fiber it would be 5ms.

 

5. People will want to play with their friends. So if you're on the Seattle Node, and your friend is on the Florida node, you can't play with each other (or if you can, you end up with hard-to-synchronize situations where you have 10ms and they have 20ms to their realm, but there's a 100ms gap between your two realms.) Now imagine someone playing from Alaska, or a 3G wireless connection in Australia, or a satellite connection in the middle of nowhere. You may be able to discount being able to play from satellite internet, but you can't cut off entire countries rural regions because you don't want to operate a realm in every city on the planet.

 

Like, I get where Google is coming from here, and I could see "serving the top 25% of internet users who live in the major cities" as being a short term goal, but what's the end goal here? I don't think the business has thought this far ahead. They probably looked at how successful MMO games are, and went "hey, what if we can make it so everyone can have the same experience and eliminate the cheat potential at the same time?" Or maybe it was as simple as "hey make your games exclusive to us, and they will never be pirated again." Like Let's Play's weren't a thing.

 

Using AI to try and predict how the player will act is likely going to have unforeseen consequences for competitive gameplay. It might work to solve jitter in latency so that a player pushing forward on their analog stick is "still pushing forward" until told to stop. Unlike traditional MMO game actions where if you drop the connection your character just stands still and gets beat up, or in some better games, has an auto-attack/defense the server will still apply even if you don't react.