Windows discrete GPU driver stability tested; but there's a catch

[Humbug]

There was a time where Nvidia had objectively better graphics drivers than the competition. ATI and later AMD developed a reputation of being second best in that regard at the time. While it is generally accepted in the enthusiast community that those days are long gone (particularly since the formation of RTG) the stigma of it still affects AMD today when it comes to buyer decisions. As can be expected it takes time for the general public who is less informed to catch up to the current state of affairs. It's something that requires more marketing from AMD which is difficult given the brand strength of Geforce.

 

QA Consultants, a reliable Candian company specializing in software testing has conducted a test of Windows graphics driver stability on modern discrete GPUs comparing AMD and Nvidia, both gaming and workstation graphics cards were included.

 

Quadro® P5000

Quadro® P4000

Quadro® P600

GeForce® GTX 1080 Ti

GeForce® GTX1060–6 GB

GeForce® GTX 1050–2 GB

Radeon™ Pro WX 9100

Radeon™Pro WX 7100

Radeon™ Pro WX 3100

Radeon™ Vega RX64

Radeon™ RX 580 8GB

Radeon™ RX 560

 

The GPUs were subjected to continuous 24x7 workloads for 12 days, and the frequency of BSODs, crashes or unexpected behavior was logged. Rather than trying to paraphrase their methodology further let me quote the below

Quote

 

We used Microsoft’s Windows HLK as a source of graphics test vectors. Windows HLK is a test framework designed to help hardware manufacturers deliver a quality product to users running Windows operating systems. Within Windows HLK, we used the 64-bit variant of CRASH to stress the Graphics Processing Unit (GPU). CRASH is a GPU stress test tool that spans 4 hours in length and captures test cases covering S3, display resolution changes, display orientation changes, content protection, and rendering.

While 4 hours of stress testing is a good indicator of prominent quality issues, it does not suffice in capturing intermittent stability failures or glitches. Therefore, we ran this test back-to-back around the clock for 12 days for each GPU. This accounts for 288 hours of non-stop stress in a test designed to make the GPU driver fail.

 

 

the full 125 page report is available below and easily  readable

https://www.amd.com/system/files/documents/graphics-driver-quality.pdf

After 12 days Nvidia GPUs passed 356 out of 436 without crashes or hangs. AMD passed 416 out of 436.

 

Now for the catch. This testing was commissioned by AMD.

 

Does that mean that the results are false? Probably not, looking at the source; a legitimate large company that does QA testing for HP, Microsoft, BMW, Apple, AT&T etc... they are not going to throw their reputation down the drain by publishing a false report.

 

But what it does mean IMO is that if Nvidia had come out on top then these results would have remained an internal AMD report and would never have been made public. AMD only published this because the results are favorable, it's possible that there are unfavorable research papers from the past comissioned by both AMD and Nvidia that they keep secret because it doesn't fit their narrative. But this time AMD is turning it into a marketing push.

 

Furthermore according to the report although GPUs were swapped between test systems to prevent bias there was no variety of test systems. All were Intel coffee lake test systems on MSI socket 1151 motherboards. Therefore I don't think this can be taken as definitive scientific proof that AMD is more stable than Nvidia across a variety of hardware systems. It will remain however a useful point of information.

 

Another thing to observe, it seems that workstation GPUs aren't any more stable than gaming GPUs, regardless of IHV (nvidia or AMD).

Cheers

[GoldenLag]

Green Army Approaches from the west------- Red Army approaches from the east

[Humbug]

1 minute ago, GoldenLag said:

Green Army Approaches from the west------- Red Army approaches from the east

just get the popcorn and sit down.

[PlayStation 2]

This is interesting.

Hopefully AMD commissioning the test didn't really affect anything. I hold that it could, but I digress.

[Froody129]

I've had plenty of issues with Nvidia drivers and I haven't even been that into PCs for very long. The drivers feel very bloated right now. 

 

I've actually never personally used an AMD card but I think we are at the point where both sides are going to provide an excellent experience. People underestimate how far we've come since CLI OS drivers  

[GoldenLag]

6 minutes ago, Humbug said:

just get the popcorn and sit down.

Let the battle of Linusgrad commence!

[Tabs]

I've used nvidia for a good while now and I have subjectively had more problems with both camps whenever I install every component of the graphics driver. I don't install the 3D Vision drivers, I only install the base graphics driver, the audio driver for when I need audio pass through, and PhysX. Nothing else.

 

I don't understand why the 3D vision driver is a default install even when no component of the system supports 3D. Even the default Windows driver downloaded from Windows Update on a fresh install installs everything. 

 

Some people might not think this is a big deal, but in the last year or so when I've been diagnosing memory leaks for people, it's hard not to notice the 3D vision driver base injected into every running process (since it's a driver, of course - but for what purpose?)

 

Anyway, I have never had any major issues on either side of the fence. I had a good experience running my older ATI cards, and even my 7970 got some good use last year when my partners machine broke and he needed a backup. I think the issue is probably overblown on both sides, outside of niche scenarios where both camps probably fare badly.

[Vode]

Nvidia had a driver that actually killed GPUs so this doesn‘t suprize me. 

 

But seriously AMDs driver development during the last 2 years has been stellar, especially with the new Radeon settings user interface. It‘s superb. 

 

They stepped up their game from the Catalyst days.

[samcool55]

After taking a quick look at the results, it's clear the pro-grade cards (quadro and radeon pro cards) failed more often than the geforce and RX cards. I would expect it would be the other way round, but nope.

 

I guess if someone complains about the results they can put their money where their mouth is and do it themselves. Everything is documented so you can do it yourself if you want to.

[Trixanity]

I'm confident the numbers are accurate.

 

However it's easy to reach a certain conclusion if you construct the tests a certain way or omit certain results.

 

Governments and corporations do it all the time. They tell their people to make the numbers work. And they'll do it. 

 

Example: city has measured a lot of air pollution from cars. City has a green profile and these numbers look bad. On top of that, city is facing potential fines because of the health and environmental concern. How do you solve that problem without drastic and costly changes? You get your best people on it and they move the measuring stations further away and now the city is no longer polluted because the numbers dropped. You're now an environmentally conscious pioneer. And you don't get fined either. It's brilliant. The best part is that the numbers are not false.

 

Just a friendly reminder to show preference to independent research.

[LAwLz]

Haven't read the full report yet, but I have 2 points to make.

1) Driver stability varies A LOT from release to release. I don't think testing a single driver from the two vendors is enough to make a generalized statement about all their drivers.

2) They only tested it with one program. Far more programs needs to be tested before you can make generalized claims. It's the same reason as why you don't just run one benchmark when testing which one of two GPUs are the best.

 

Not to mention the issues with the tests being funded by AMD, as explained in the OP and by @Trixanity.

[Humbug]

22 minutes ago, Vode said:

But seriously AMDs driver development during the last 2 years has been stellar, especially with the new Radeon settings user interface. It‘s superb. 

Agreed. You can see the massive turn most visibly in the Linux world since the formation of RTG, cause the drivers there were a turd before. The Linux community went from only recommending Nvidia stuff to now a consensus that AMD's open driver model is a better fit and better integrated with the kernel so it just works out of the box with good performance. And the driver devs themselves post on forums regularly engaging with the community unlike any other IHV. Even getting into arguments lol.

[Dabombinable]

4 minutes ago, LAwLz said:

Haven't read the full report yet, but I have 2 points to make.

1) Driver stability varies A LOT from release to release. I don't think testing a single driver from the two vendors is enough to make a generalized statement about all their drivers.

2) They only tested it with one program. Far more programs needs to be tested before you can make generalized claims. It's the same reason as why you don't just run one benchmark when testing which one of two GPUs are the best.

 

Not to mention the issues with the tests being funded by AMD, as explained in the OP and by @Trixanity.

Some games used to occasionally cause a driver crash with my GTX 970 at its stock clocks, while others would be just fine - even with a normally unstable 100MHz overclock. That's how erratic driver crashes can be and why far more testing that isn't being backed by one of the tested companies is needed to even get close to accurate results.

[Humbug]

16 minutes ago, Trixanity said:

I'm confident the numbers are accurate.

 

However it's easy to reach a certain conclusion if you construct the tests a certain way or omit certain results.

 

Governments and corporations do it all the time. They tell their people to make the numbers work. And they'll do it. 

 

Example: city has measured a lot of air pollution from cars. City has a green profile and these numbers look bad. On top of that, city is facing potential fines because of the health and environmental concern. How do you solve that problem without drastic and costly changes? You get your best people on it and they move the measuring stations further away and now the city is no longer polluted because the numbers dropped. You're now an environmentally conscious pioneer. And you don't get fined either. It's brilliant. The best part is that the numbers are not false.

 

Just a friendly reminder to show preference to independent research.

Agreed. For example it remains a possibility that AMD is a bit more stable on coffee lake systems so they commissioned research only on that. Or maybe they thought that particular month's Nvidia driver is buggier than usual so let's test now. We will not know for sure...

[Taf the Ghost]

It should be noted that the Driver issues flipped around the time it went from ATI to AMD graphics. Nvidia has had a lot of driver issues since the late 2000s.

[Arika]

I've never had any drivers on any GPU i'v had. Is it a big enough issue than AMD felt the need to fund this kind of test? it does raise a few questions though

 

• What was AMD hoping to get out of this?
• Was NVidia aware of such an upcoming test to optimize their driver's as ADM might have had the time to do.
• Because i dont feel like reading the full report, do they go into detail about what kind of workload was running at the time of the failure? ie a workstation load may affect gaming cards in a different way
• How is AMD going to prove that this is an unbias test when it makes them look better than the competition and they funded it (like when people got up in arms about the AMD flaw that they though was an intel smear campaign funded by them)

[asus killer]

having both i think AMD shit works best. But come on studies paid for by an interested party seem crazy, to me i can't stop but think the customer is always right.

[TetraSky]

I've seen more Nvidia drivers issues in the news in the past year than I've seen AMD issues, so I don't find this hard to believe.

All I can say to the mindless drones defending Nvidia on this... Just because you don't have any issues, doesn't mean it's the same for everyone else. Same for AMD.

 

I've had issues with both AMD and NVIDIA, especially on older cards that claim to still be supported yet the latest drivers aren't working for shit, only the archived ones do...

Or Windows 10 installing faulty graphic drivers on an older computer (9 years old, Nvidia) and laptop(8 years old, AMD) when they clearly don't work properly(videos don't play, only audio works, same issues on both, funny that) with no way to prevent their installation. I tried EVERYTHING possible on Win10 Home, the only thing that worked was disabling Windows Update... then it fucking re-enabled itself, somehow, and installed the bad drivers again... ugh, This is my only major complain about drivers and it's not even related to AMD or NVIDIA, it's microsoft pushing bad drivers on us.

[leadeater]

Why is it that all the professional cards form both AMD and Nvidia had far higher issues? Aren't we paying more for less of those????

[vanished]

5 hours ago, leadeater said:

Why is it that all the professional cards form both AMD and Nvidia had far higher issues? Aren't we paying more for less of those????

No, you're paying more because you can.  That's how business works

[DrMacintosh]

11 hours ago, GoldenLag said:

east

Oooooh, East? I thought you said Weast

[DrMacintosh]

AMD needs some Fife & Drum for their slow march back to the top

 

[cj09beira]

9 hours ago, Arika S said:

I've never had any drivers on any GPU i'v had. Is it a big enough issue than AMD felt the need to fund this kind of test? it does raise a few questions though

 

• What was AMD hoping to get out of this?
• Was NVidia aware of such an upcoming test to optimize their driver's as ADM might have had the time to do.
• Because i dont feel like reading the full report, do they go into detail about what kind of workload was running at the time of the failure? ie a workstation load may affect gaming cards in a different way
• How is AMD going to prove that this is an unbias test when it makes them look better than the competition and they funded it (like when people got up in arms about the AMD flaw that they though was an intel smear campaign funded by them)

there are many people out there with big misconceptions about amd, one of which is that their drivers suck so they are trying to show thats not the case 

[mr moose]

You can't kill internet narrative with a half arsed study into drivers.   I would have thought AMD would have learnt by now, the only thing that will kill a bad reputation on the internet is better performance and cooler PR.  

[DrMacintosh]

11 minutes ago, mr moose said:

You can't kill internet narrative with a half arsed study into drivers.   I would have thought AMD would have learnt by now, the only thing that will kill a bad reputation on the internet is better performance and cooler PR.  

AMDs PR has been pretty cool lately

 

Look at that Intel jab with the "more advanced security features"