Ex nVidia engineer: every game ships broken, we fix it in drivers

[patrickjp93]

They are not lazy. They must produce a game each year with priority for consoles and then they have less time to optimize for PC... Publishers (EA, Ubi etc.) are money hungry bastards in this case

Which is weird, because there are far more pc gamers than console gamers.

[Thony]

Which is weird, because there are far more pc gamers than console gamers.

Not really true. Nowadays some people have both so it increases console count.

And with the sale figures publishers get from console players compared to PC they have no reason to focus on PC any more than they do already.

We often buy games from other sources or wait for discounts whereas console owners pay full price most of the time.

[BobbyG2]

Which is weird, because there are far more pc gamers than console gamers.

But there are also far more PC games than console games.

Console games make more on average than PC games (per title) simply because there's far less games to choose from, and subsequently a far lower standard of quality.

In order to make a lot of money from a PC game, you have to be able to stand out from the crowd. And on an open platform, where the sky is the limit, and there's no one to answer to for what's in your game besides your customers, it's far harder to rise above everyone else.

[7850OC]

Games ship broken, we know it. But who do you blame when your shiny powerful GPU performs like rubbish, or shows tons of geometry and texture glitches? In many cases the game code is so broken that hardware manufacturer needs to step in. And apparently this had become so common practice now, that game developers do not bother to optimize, or even properly write code running on GPUs at all, because they know "it will get fixed in drivers". Per forum post on gamedev.net:

 

 

This is also one of the reasons why it is so hard for smaller manufacturers of graphic chips to break into this market: they just don't have resources to cope with the amount of 'gaming history' built into drivers, and don't have engineering capacities able to comprehend both source code of drivers, and game itself.

 

 

Another interesting observation, in the light of recent new APIs that we all look forward to:

 

 

Source: http://www.gamedev.net/topic/666419-what-are-your-opinions-on-dx12vulkanmantle/#entry5215019

 

 

I did not mean to bash on game developers. Heck, I can't program myself and I have very deep respect to people that can do beautiful and complex programs such as video games. I know many times they work close with AMD or nVidia to get the games running in best possible way. I just did not know it had grown into this proportions. Optimizing for certain hardware functions is one thing. Relying on hardware manufacturers to solve my own convenience and screw ups? That's another.

 

 

What's the difference between an intern and regular engineer working there? I am sorry if I messed up, I am not from English speaking country.

 

great post, welcome to the forum new friend 

[Zulkkis]

Lazy is the wrong kind of a word, resources is more like it.

 

And if I read the text correctly, it's more about the APIs being a complete mess that not too many people can handle correctly. I mean, come on, would all "Ubisoft / EA / Valve / Epic / iD" etc. be having these issues if it was really just a question of "laziness"? Hell it already says that futuremark (they make pretty much 3D mark only these days) can't get more than 5% gains in their benchmark that's not even a game, what can you expect game developers or nVidia / AMD to get? We're talking of guys with the talent that it only takes two of them to make an engine of this caliber on their own.

[typographie]

This just proves how lazy developers are.

 

Except that it isn't proof, more of an anecdote.

 

Assuming this is true I think it's still an open question how much the developers themselves are to blame. We've been hearing from ex-game developers for years who complain that these decisions are being made by the executives. Too much spent on marketing and over-ambitious deadlines resulting in far, far too few development resources.

[Prastupok]

I'd recommend everyone read the post, which isn't reliable to begin with... not just the quoted bits...

[7850OC]

here is 

 

I'd recommend everyone read the post, which isn't reliable to begin with... not just the quoted bits...

 

Promit - Posted 06 March 2015 - 02:56 PM

 

Many years ago, I briefly worked at NVIDIA on the DirectX driver team (internship). This is Vista era, when a lot of people were busy with the DX10 transition, the hardware transition, and the OS/driver model transition. My job was to get games that were broken on Vista, dismantle them from the driver level, and figure out why they were broken. While I am not at all an expert on driver matters (and actually sucked at my job, to be honest), I did learn a lot about what games look like from the perspective of a driver and kernel.

 

The first lesson is: Nearly every game ships broken. We're talking major AAA titles from vendors who are everyday names in the industry. In some cases, we're talking about blatant violations of API rules - one D3D9 game never even called BeginFrame/EndFrame. Some are mistakes or oversights - one shipped bad shaders that heavily impacted performance on NV drivers. These things were day to day occurrences that went into a bug tracker. Then somebody would go in, find out what the game screwed up, and patch the driver to deal with it. There are lots of optional patches already in the driver that are simply toggled on or off as per-game settings, and then hacks that are more specific to games - up to and including total replacement of the shipping shaders with custom versions by the driver team. Ever wondered why nearly every major game release is accompanied by a matching driver release from AMD and/or NVIDIA? There you go.

 

The second lesson: The driver is gigantic. Think 1-2 million lines of code dealing with the hardware abstraction layers, plus another million per API supported. The backing function for Clear in D3D 9 was close to a thousand lines of just logic dealing with how exactly to respond to the command. It'd then call out to the correct function to actually modify the buffer in question. The level of complexity internally is enormous and winding, and even inside the driver code it can be tricky to work out how exactly you get to the fast-path behaviors. Additionally the APIs don't do a great job of matching the hardware, which means that even in the best cases the driver is covering up for a LOT of things you don't know about. There are many, many shadow operations and shadow copies of things down there.

 

The third lesson: It's unthreadable. The IHVs sat down starting from maybe circa 2005, and built tons of multithreading into the driver internally. They had some of the best kernel/driver engineers in the world to do it, and literally thousands of full blown real world test cases. They squeezed that system dry, and within the existing drivers and APIs it is impossible to get more than trivial gains out of any application side multithreading. If Futuremark can only get 5% in a trivial test case, the rest of us have no chance.

 

The fourth lesson: Multi GPU (SLI/CrossfireX) is fucking complicated. You cannot begin to conceive of the number of failure cases that are involved until you see them in person. I suspect that more than half of the total software effort within the IHVs is dedicated strictly to making multi-GPU setups work with existing games. (And I don't even know what the hardware side looks like.) If you've ever tried to independently build an app that uses multi GPU - especially if, god help you, you tried to do it in OpenGL - you may have discovered this insane rabbit hole. There is ONE fast path, and it's the narrowest path of all. Take lessons 1 and 2, and magnify them enormously. 

 

Deep breath.

 

Ultimately, the new APIs are designed to cure all four of these problems.

* Why are games broken? Because the APIs are complex, and validation varies from decent (D3D 11) to poor (D3D 9) to catastrophic (OpenGL). There are lots of ways to hit slow paths without knowing anything has gone awry, and often the driver writers already know what mistakes you're going to make and are dynamically patching in workarounds for the common cases.

* Maintaining the drivers with the current wide surface area is tricky. Although AMD and NV have the resources to do it, the smaller IHVs (Intel, PowerVR, Qualcomm, etc) simply cannot keep up with the necessary investment. More importantly, explaining to devs the correct way to write their render pipelines has become borderline impossible. There's too many failure cases. it's been understood for quite a few years now that you cannot max out the performance of any given GPU without having someone from NVIDIA or AMD physically grab your game source code, load it on a dev driver, and do a hands-on analysis. These are the vanishingly few people who have actually seen the source to a game, the driver it's running on, and the Windows kernel it's running on, and the full specs for the hardware. Nobody else has that kind of access or engineering ability.

* Threading is just a catastrophe and is being rethought from the ground up. This requires a lot of the abstractions to be stripped away or retooled, because the old ones required too much driver intervention to be properly threadable in the first place.

* Multi-GPU is becoming explicit. For the last ten years, it has been AMD and NV's goal to make multi-GPU setups completely transparent to everybody, and it's become clear that for some subset of developers, this is just making our jobs harder. The driver has to apply imperfect heuristics to guess what the game is doing, and the game in turn has to do peculiar things in order to trigger the right heuristics. Again, for the big games somebody sits down and matches the two manually. 

 

Part of the goal is simply to stop hiding what's actually going on in the software from game programmers. Debugging drivers has never been possible for us, which meant a lot of poking and prodding and experimenting to figure out exactly what it is that is making the render pipeline of a game slow. The IHVs certainly weren't willing to disclose these things publicly either, as they were considered critical to competitive advantage. (Sure they are guys. Sure they are.) So the game is guessing what the driver is doing, the driver is guessing what the game is doing, and the whole mess could be avoided if the drivers just wouldn't work so hard trying to protect us.

 

So why didn't we do this years ago? Well, there are a lot of politics involved (cough Longs Peak) and some hardware aspects but ultimately what it comes down to is the new models are hard to code for. Microsoft and ARB never wanted to subject us to manually compiling shaders against the correct render states, setting the whole thing invariant, configuring heaps and tables, etc. Segfaulting a GPU isn't a fun experience. You can't trap that in a (user space) debugger. So ... the subtext that a lot of people aren't calling out explicitly is that this round of new APIs has been done in cooperation with the big engines. The Mantle spec is effectively written by Johan Andersson at DICE, and the Khronos Vulkan spec basically pulls Aras P at Unity, Niklas S at Epic, and a couple guys at Valve into the fold.

 

Three out of those four just made their engines public and free with minimal backend financial obligation.

 

Now there's nothing wrong with any of that, obviously, and I don't think it's even the big motivating raison d'etre of the new APIs. But there's a very real message that if these APIs are too challenging to work with directly, well the guys who designed the API also happen to run very full featured engines requiring no financial commitments. So that's served to considerably smooth the politics involved in rolling these difficult to work with APIs out to the market.

 

The last piece to the puzzle is that we ran out of new user-facing hardware features many years ago. Ignoring raw speed, what exactly is the user-visible or dev-visible difference between a GTX 480 and a GTX 980? A few limitations have been lifted (notably in compute) but essentially they're the same thing. MS, for all practical purposes, concluded that DX was a mature, stable technology that required only minor work and mostly disbanded the teams involved. Many of the revisions to GL have been little more than API repairs. (A GTX 480 runs full featured OpenGL 4.5, by the way.) So the reason we're seeing new APIs at all stems fundamentally from Andersson hassling the IHVs until AMD woke up, smelled competitive advantage, and started paying attention. That essentially took a three year lag time from when we got hardware to the point that compute could be directly integrated into the core of a render pipeline, which is considered normal today but was bluntly revolutionary at production scale in 2012. It's a lot of small things adding up to a sea change, with key people pushing on the right people for the right things.

 

 

Phew. I'm no longer sure what the point of that rant was, but hopefully it's somehow productive that I wrote it. Ultimately the new APIs are the right step, and they're retroactively useful to old hardware which is great. They will be harder to code. How much harder? Well, that remains to be seen. Personally, my take is that MS and ARB always had the wrong idea. Their idea was to produce a nice, pretty looking front end and deal with all the awful stuff quietly in the background. Yeah it's easy to code against, but it was always a bitch and a half to debug or tune. Nobody ever took that side of the equation into account. What has finally been made clear is that it's okay to have difficult to code APIs, if the end result just works. And that's been my experience so far in retooling: it's a pain in the ass, requires widespread revisions to engine code, forces you to revisit a lot of assumptions, and generally requires a lot of infrastructure before anything works. But once it's up and running, there's no surprises. It works smoothly, you're always on the fast path, anything that IS slow is in your OWN code which can be analyzed by common tools. It's worth it.

[cesrai]

I'm pretty sure they both get paid for those stupid "nvidiaaaaa" and "AMD" logos when you launch games. Granted is not much since there's no true exclusivity (other than a few features like whatever flavour of hair physics or AA you might prefer) but it's something so it's not a completely shit on em for free situation.

Actually tressfx is open source.

[Mew]

Nvidia and AMD should just publicly shame developers already. What do they have to lose? Nothing. Lazy developers hurt gaming more than anything else. Publishers are greedy, we all know that, but a heartless developer who doesn't give a shit and keeps pumping out mediocrity that flat out doesn't work well is even worse for this industry.

I would say that's not the extent of it either. Developers on mobile are also seemingly as lazy at optimizing performance and efficiency of their apps. Applications such as Norton and Mcafee are both known to be rather harmful for your system in terms of performance. Every web browser in Windows imo, feels bloated and unoptimized in one way or another. With no strict guidelines and discipline, most developers and programmers are just becoming sloppy.

[Arreat]

I believe this completely. I don't think it's game devs though but rather game publishers telling the studio to not bother with optimization. It also means that is why we always get new drivers every time a new AAA game comes out because we do. 

 

That's exactly right . I can't believe how many people mix up "developers" and "publishers" . It's not the dev's fault when they get the giant task of optimizing the PC game and the publisher gives them 1/3 of the team and 1/5 of the time , console versions have . 

Looking at a broken code might let you to believe , the dev is lazy , but what if this dev had to grab the code from the console version , convert it for Pc for a minimal amount of time , very near to the deadline , and at the exact date , publishers lifts what he's done and plop's it on the table for consumer to buy . 

[AHaskin14]

That was all very interesting and believable. What would happen though if you got a new game and simply never updated your drivers though?

[Tonylu1595]

That was all very interesting and believable. What would happen though if you got a new game and simply never updated your drivers though?

Step 1: Run game

Step 2: Complain about low fps in forums

Step 3: Get told to update drivers

Step 4: Update drivers

[FakezZ]

This just proves how lazy some developers are.

There. Fixed now

[szomeret]

Every game ships broken because the graphics APIs are a big mess and game developers, even if running through QC, still only validate the setup based on visual output on test machines. If they don't have enough test machines, they'll have issues. Many games abuse the OpenGL or D3D spec in someway, but never realize it because all their test machines operate the same. You really need a plethora of machines with different driver versions to notice. Some drivers completely ignore the spec anyway (INTEL!!).

[azurezeed]

if true that explains a lot

[euniqe]

hmmm...

 

 

 

He's an "ex" engineer....

 

 

Take this with a grain of salt people. However, it does seem to be extremely likely.

[Prastupok]

He's an "ex" engineer....

 

 

He was an "ex" Nvidia intern.

[AHaskin14]

Step 1: Run game

Step 2: Complain about low fps in forums

Step 3: Get told to update drivers

Step 4: Update drivers

Well I guess that would be the process.

[SanityAgathion]

Lazy is the wrong kind of a word, resources is more like it.

 

And if I read the text correctly, it's more about the APIs being a complete mess that not too many people can handle correctly. I mean, come on, would all "Ubisoft / EA / Valve / Epic / iD" etc. be having these issues if it was really just a question of "laziness"? Hell it already says that futuremark (they make pretty much 3D mark only these days) can't get more than 5% gains in their benchmark that's not even a game, what can you expect game developers or nVidia / AMD to get? We're talking of guys with the talent that it only takes two of them to make an engine of this caliber on their own.

 

Actually, that's mostly what I got from the post, too. I do not think it's developer's fault all the time, many issues that drivers fix are from using dirty tricks or simply lack of knowledge due to complexity of underlying APIs.

 

great post, welcome to the forum new friend 

 

 Thanks

[Ertman]

Intel can't keep up with the necessary investment? What?

[vm'N]

Intel can't keep up with the necessary investment? What?

Big corporations is split up into departments, which will be assigned a budget.

It is not one single department with a single budget.

[Zulkkis]

Intel can't keep up with the necessary investment? What?

 

The other post about APIs that has been flying around explains it a bit better. Intel doesn't really want to make GPUs, but it kind of ties into their business and they have the die size to spare, and nothing better to do with it.

 

Essentially there's not enough ROI for Intel to force themselves into the high-end discrete GPU market. They just offer CPUs and integrated solutions.

[mr moose]

Didn't we kinda already know this?   I mean since the mid 90s there are only three companies that ship properly finished and tested software (adobe, MS and Apple (OS's are a little more tricky which is why there are always updates and patches)).  everyone else just sells you the basic thing and expects the consumer to be the final part of QC. 

 

Although to be honest I am surprised that they can fix so much dodgy and unoptimized game code with drivers. To me that still sounds like a bit of a stretch.

[dimitriianghelov]

why not just blame microsoft and for their dx 9/11