Update: Game now patched | Rockstar Games confirms major reductions to loading times coming to GTA Online thanks to a GTA modder

[Levent]

I just got a 300mb+ update in steam (1.54) and it seems it took my online load screen from 5+ minutes to down to 2. Can someone else confirm?

[PorkishPig]

26 minutes ago, Levent said:

I just got a 300mb+ update in steam (1.54) and it seems it took my online load screen from 5+ minutes to down to 2. Can someone else confirm?

Looks like a launcher patch.

https://steamdb.info/depot/271591/

https://steamdb.info/depot/271592/

 

Nope, never mind. Can confirm this was the patch. Just got 35 second load time from Story Mode.

https://twitter.com/GTASeries/status/1371792213999562754

Edited March 16, 2021 by PorkishPig

[Silentprototipe]

2 hours ago, James Evens said:

@Silentprototipe Is it worse or on pair compared to his original patch?

About 5 seconds off the original creator's patch. R* practically bought that guy's code. 

 

28 minutes ago, WereCat said:

I stopped playing it because of that... and the awful UI. 

But I will give it another try once the update goes live. 

Lol dont, Ive racked up close to 3.5k hours in the past year and I would recommend people stay the hell away from it 

[Doobeedoo]

Nice.

[Vishera]

7 hours ago, Loote said:

Nintendo would probably sue him and do their best to make the patch impossible, props to Rockstar for giving him some compensation at least.

Rockstar are actually very chill towards modders (as long as you don't promote piracy/harm their online cash flow)

As a modder of GTA games myself i like the messages that rock star leaves in their source code:

; Author: Bill Henderson
; Created 10/12/1999
;
; > COMMENTS <
; ------------
; TWEAK AT YOUR OWN RISK
;
; > UNITS <
; ---------
; vehicle identifier 14 characters max
; dimensions in metres
; mass in Kg
; velocity in Km/h
; acceleration/deceleration in ms-2
; multipliers x1.0 is default
; angles in degrees
; Levels :- (L)ow, (M)edium, (H)igh
; buoyancy = percent submerged (> 100% vehicle sinks)
; engine type :- (P)etrol, (D)iesel, (E)lectric
; engine drive :- (F)ront, (R)ear, (4)-wheel drive

That's from the source code of GTA San Andreas,this is part of a code that is responsible for vehicle handling,

With detailed explanation by the programmer (Thanks Bill Henderson!) of what everything does.

 

By the way,Bill Henderson is now working at Epic Games.

[kvuj]

5 hours ago, RejZoR said:

When some random Anon on the interwebs makes loading times 70% shorter. Something that makers of the game and game engine itself couldn't somehow. I HAVE QUESTIONS...

Not only that, but without the source code, and with pure reverse engineering.

 

Just goes to show how optimizing hot code paths is always a great idea (it was a simple parsing inefficiency + comparing every single entry vs using a hash map). I guess loading time is less important than polish when you're in the last crunch before release but still. 

 

Also could the author please link the amazing blog post by the hacker? (https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/)

[Silentprototipe]

2 hours ago, James Evens said:

There was not much to code. Either they just took it or let the intern write it during the coffee break.

 

Largest part of this was getting to the point which causes the problem. I still don't get why nobody at rockstar ever though about doing it. They even have the source code.

 

Rockstar are a bunch of apes in a coding warehouse when it comes to patching anything in this game. So many day 1 glitches that ive seen that are still in the game despite being well known. This time however it was so blatently bad that R* finally did something 

[Beskamir]

23 hours ago, AluminiumTech said:

Rockstar Games have said they will incorporate the fix into a new update coming soon to GTA Online and have compensated the modder for his work. 

Wow that's really unexpected!

 

  

12 hours ago, BuckGup said:

So everyone working at Rockstar never took a Computer Science class in their life? 

The people that did had more important things to work on. This was clearly done by the marketing "devs" that probably shouldn't even be allowed near a computer.

[Beskamir]

Realized I was double posting mods pls delete.

[wanderingfool2]

12 hours ago, BuckGup said:

So everyone working at Rockstar never took a Computer Science class in their life? 

 

Well, I think it was probably one person who implemented things (likely with a smaller JSON file) and no one decided to check what was causing the long load times after development.  After all, the big mistake was pretty simple, that I think a decent amount of CS people would miss.  Using sscanf multiple times on a 12MB string...the issue is that sscanf essentially does something similar to strlen which scans the entire string (despite the fact that logically sscanf shouldn't need to know the string length).

 

The hashmap one, I can't really make any excuses there though...especially given that they only appear to use it on the single JSON file (by the sounds of it) and there aren't any duplicate entries in the JSON file.

[AlTech]

17 hours ago, PorkishPig said:

Looks like a launcher patch.

https://steamdb.info/depot/271591/

https://steamdb.info/depot/271592/

 

Nope, never mind. Can confirm this was the patch. Just got 35 second load time from Story Mode.

https://twitter.com/GTASeries/status/1371792213999562754

 

17 hours ago, Levent said:

I just got a 300mb+ update in steam (1.54) and it seems it took my online load screen from 5+ minutes to down to 2. Can someone else confirm?

It looks like it has been added in GTA Online Title Update 1.53 with Rockstar providing credit where credit is due in the patch notes . I’ve updated the Original post with this as well.

 

https://support.rockstargames.com/articles/360061161574/GTAV-Title-Update-1-53-Notes-PS4-Xbox-One-PC

[Video Beagle]

17 hours ago, RejZoR said:

When some random Anon on the interwebs makes loading times 70% shorter. Something that makers of the game and game engine itself couldn't somehow. I HAVE QUESTIONS...

To solve a problem, you not only need to be looking to solve it and be looking in the right place. Loading screen time was likely a very low priority problem.. It's loading? Good, next issue....  And lots of times people very familiar with the system like the makers don't think to look at what can be obvious to an outsider. Especially if the system they're using works in other games as is.

[RejZoR]

Just now, Video Beagle said:

To solve a problem, you not only need to be looking to solve it and be looking in the right place. Loading screen time was likely a very low priority problem.. It's loading? Good, next issue....  And lots of times people very familiar with the system like the makers don't think to look at what can be obvious to an outsider. Especially if the system they're using works in other games as is.

It's not a low priority when you're sitting around for 5 minutes on high(er) end systems. Like, come on?

[mariushm]

For those that don't know how this fix reduced the loading time...

 

There were two problems.

 

The game downloads a JSON file (kind of a TEXT file with a specific structure) with around 60 k of items available in game and their price and some information about them, every time you go online.  That file is currently UP to around 10 MB in size.

When they made the game and tested the code, there weren't so many game items, they probably had only a few hundred, and maybe less than 1 MB in size for that file. 

 

The game code uses a single cpu core, a single thread, to parse (to read the json file sequentially) and take every game item and its information and put it in memory in a list of items. 

They probably used a ready made JSON parsing library (some code made by someone else) which was badly written, as in every time a small piece of information from the JSON file was decoded  (a property of a game item), that library was calling a function to find out the whole size of the json file .. so for each of those 60k game items, the library was asking how big the file is maybe 10-20 times, and that function took time to actually determine how big the file is. 

 

So it's quite possible when they first made and released the game, the problem wasn't noticeable, because the third party library worked well with small JSON files .... but it's increased exponentially with very big files.

And by that time it got seriously slow, probably developers already moved to making red dead redemption or this was considered a low priority bug  or a bug that wasn't worth releasing a full patch / update (microsoft and sony made you pay to release updates if you did too many) 

 

 

One a game item and its properties were extracted from the JSON file, it's put in a list. But, the game code actually checks all the previously game items added to the list to see if the game item wasn't already added to the list.

So for 2nd item, game checks item 1 and compares with the new item. For 3rd item .. game checks items 1 and 2 already in the list. For 100 item, game checks items 1 to 99 ... so you can see that when you get to game item 63,000  ... you check items 1 to 63000 .... so during this load the game keeps checking millions of times if each game item was added to the list already or not

 

They could have fixed the game in maybe 1 hour by changing from json to a different format which can be parsed without having to determine file lengths and crap like that. 

They could have used hashes or unique identifiers for each game item so that when a new game item is added, the code doesn't have to check all previous 60k game items to see if they're unique or not. 

 

[Video Beagle]

4 minutes ago, mariushm said:

That file is currently UP to around 10 MB in size.

When they made the game and tested the code, there weren't so many game items, they probably had only a few hundred, and maybe less than 1 MB in size for that file.

I was wondering if it was something like that....a function that was fine in the early days but as the game proceeded was uneffecient for the task.

[RejZoR]

That's just dumb design which you certainly wouldn't expect from AAA developer like Rockstar...

[Master Disaster]

6 minutes ago, Video Beagle said:

To solve a problem, you not only need to be looking to solve it and be looking in the right place. Loading screen time was likely a very low priority problem.. It's loading? Good, next issue....  And lots of times people very familiar with the system like the makers don't think to look at what can be obvious to an outsider. Especially if the system they're using works in other games as is.

Thats called creation/creator bias and yep, its a HUGE issue in loads of different areas.

 

Level design is one example, the person that created the level already knows the intended solution since they created the problem and while testing will, more often than not, miss glaringly obvious problems (these days commonly called cheese) simply because they're focus is entirely on whether the intended solution works as they intended it to.

 

This is why level creation teams will spend hours play testing each others levels and get people from outside the team to also play them. The more eyes you can get on something before it goes live the more chance you have of picking up problems.

[Video Beagle]

1 minute ago, Master Disaster said:

Thats called creation/creator bias and yep, its a HUGE issue in loads of different areas.

Hell, you see it in non technical things with just proof reading. You know what you meant to type, so you don't see the error no matter how many times you go over it.  (I've had that from both the writer and the editor's side :D) .

 

[Kisai]

On 3/17/2021 at 1:42 AM, mariushm said:

For those that don't know how this fix reduced the loading time...

 

There were two problems.

 

The game downloads a JSON file (kind of a TEXT file with a specific structure) with around 60 k of items available in game and their price and some information about them, every time you go online.  That file is currently UP to around 10 MB in size.

When they made the game and tested the code, there weren't so many game items, they probably had only a few hundred, and maybe less than 1 MB in size for that file. 

 

 

Sounds like the problem was that they used JSON at all. It was probably generated from some php or node.js script on the server.

 

JSON is Fine when you deal largely with one-thing-per-file. It’s awful as an an analog for a data table, because you don’t have any of the features that even SQLite has.

 

it’s likely all that was needed to fix this loading problem was to rewrite the parsing library to load the file into RAM, then do operations on it rather than one key pair at a time.

 

I’ve seen similar problems with games ported from CJK languages, where because English is three times longer to parse, the rendering code was dragging down the game anytime there was text on the screen.

 

TEXT processing tends to be the thing programmers are naive about.

[mariushm]

8 minutes ago, Kisai said:

Sounds like the problem was that they used JSON at all. It was probably generated from some php or node.js script on the server.

 

JSON is Fine when you deal largely with one-thing-per-file. It’s awful as an an analog for a data table, because you don’t have any of the features that even SQLite has.

 

it’s likely all that was needed to fix this loading problem was to rewrite the parsing library to load the file into RAM, then do operations on it rather than one key pair at a time.

 

I’ve seen similar problems with games ported from CJK languages, where because English is three times longer to parse, the rendering code was dragging down the game anytime there was text on the screen.

 

TEXT processing tends to be the thing programmers are naive about.

Or just have the game servers serve the JSON file in an alternative format  - like for example UBJSON  : https://ubjson.org/

 - and then patch the game to request the new format and fall back to json if new format doesn't work ... unpatched games can continue to request json.

 

UBJSON 100% reversible but has the benefit of not having to parse text and count lengths  because you have the lengths specified for each thing ...

for example

{
  "login": "octocat",
  "id": 1,
  "avatar_url": "https://github.com/images/error/octocat_happy.gif",
  "gravatar_id": "somehexcode",
  "url": "https://api.github.com/users/octocat",
  "name": "monalisa octocat",
  "company": "GitHub",
  "blog": "https://github.com/blog",
  "location": "San Francisco"
 }

 

You need to skip everything until you find the first " , then go character by character until the next " , then look for  " : " , then parse the values etc ... that's why those scanfs and whatever else keep checking the string length...

 

The above looks like this in ubjson ( where each number between [ ]  is 1 byte, 2 bytes or 4 bytes depending on data type specified in front of that.

 

[{]
    [i][5][login][S][i][7][octocat]
    [i][2][id][i][1]
    [i][10][avatar_url][S][i][49][https://github.com/images/error/octocat_happy.gif]
    [i][11][gravatar_id][S][i][11][somehexcode]
    [i][3][url][S][i][36][https://api.github.com/users/octocat]
    [i][4][name][S][i][16][monalisa octocat]
    [i][7][company][S][i][6][GitHub]
    [i][4][blog][S][i][23][https://github.com/blog]
    [i][8][location][S][i][13][San Francisco]
[}]

 

and if someone is paranoid, they could add a CRC16 or CRC32 after every record or at the end of the whole ubjson ( a 10 MB file with 63300 records ... around 160-180 characters per record.