I'm examining Nai's benchmark and possibly making my own

[Syntaxvgm]

It's been a long time since I've done anything related to stuff like this.
I would like to start by saying, I'm not in any way an NVIDIA fanboy. Despite the hat the cat is wearing in my avatar (It's a joke because of this NVIDIA stuff)
The reason I bought a 970 was price/performance, and the fact that I am turned on sexually by nvidia reference coolers. (I spent an extra 50$ to get a reference one, a bit rare.)
 
 
Some background on why I'm starting to look at Nai's tool funny

-Other cards like the TITAN show a dip in speed towards the end
-The results I got were not really consistent with other testing I did
-He made this in visual studio 2010 for some reason. Not anything wrong with it, I just have a hatred for it that's really not justified.
-This test measures the VRAM appropriately in  Mebibytes, which is correct if you accept the decimal system made by IEC and pushed by IEEE. However, the transfer speed is measured in Gigabytes/Second instead of Gibibytes/Second. This is fine given you acknogledge the difference, but I don't know exactly how this measurement is made until I look at the source. What worries me is this was made with Visual 2010, a MS tool. MS does not accept the new decimal system pushed by IEEE, and a Gigabyte on Windows is a Gibibyte in the new system. This is why your hard disks show up smaller in windows, and is the source of many lawsuits around 2005. I don't work with visual, but given they used some sort of windows based function to measure this, it would be wrong in this context. I don't know until I look at it, and if so this is easily corrected. 
-I would like to make an AMD benchmark
-I would like to either confirm on deny this once and for all.
 
And finally, the big one
-The numbers I got changed based on how much PCIe bandwidth the card had.
I used a single channel memory configuration to test this initially. Both of these test are done as Nai had intended with the screen being connected to the iGPU instead of the 970.
This is the reason I really want to mess with this. This should NOT happen. 

 
 
When I originally discovered this issue, when I bought the card in December, I noticed through tests in UE3 that around 3.2GiB the performance tanked, but I realism now my methods for testing this were lax, and I made some mistakes regarding the testing. I just basically put a stupid amount of the same extremely high resolution texture on screen over time (the textures all treated as separate textures even though they were a copy/paste) to give me an idea of how much I had to work with. I did it for fun honestly, I hadn't gotten a new GPU for myself in a personal gaming system since a 1GB one I had and my other 2GB ones. The other thing is I noticed my troubles were not around 3500MiB, but closer to 3200. I had 5 screens connected, 4 on the 970. 
I didn't realize how much memory windows seemed to take for this, as I had the same setup on a 1GB card in the past and it didn't use much. 
After some research into WDDM and NVIDIA drivers, I found some interesting things that may explain of some of the in game benchmarks. 
 
So I believed this until I saw some other benchmarks people did in games. Some people made asinine arguments against this, but some people had some points. 
 
So what will I be doing? 
I will do a bunch of practical benchmarks as well as look at the source fro NAIs benchmark, and write my own benchmark. As a lot of you know, I have promised to do some tutorials that have not seen the light of day yet. I am very, very busy and life has been rough. But I promise this will be priority one for me. I intend to correct this benchmark if it is wrong, or make a new one if necessary. I would like to make an AMD version as well. 
 
I will either come back confirming the NVIDIA issue, which I am leaning toward, or come back disproving it. 
Wish me luck, and if anyone else thinks they could contribute, please go take a look at the source code Nai posted.

 

Also, I wasn't comfortable with posting this in the News section until someone else confirm this. Do whatever you feel is appropriate mods. 

[SuperPug]

Fuckin' Gibibytes... always makes me giggle.

[Syntaxvgm]

Fuckin' Gibibytes... always makes me giggle.

The main issue is still the PCIe bandwidth changing the numbers on the tail end

[SuperPug]

The main issue is still the PCIe bandwidth changing the numbers on the tail end

 

Yeah that is odd... But regardless of whether this initial testing mehtodology was flawed or not, the realworld performance impact still strongly depends on Nvidias drivers anyway. If they manage to handle the memory allocation correctly (less frequently accessed resources on the low bandwith segment) the ''bottleneck'' shouldn't matter much either way.

[SuperPug]

-

[Razzaa]

Nai posted on FB that his benchmark is flawed. He confirmed this last week i believe.

[Razzaa]

Edit: Removed because wrong quote

[Syntaxvgm]

Hey @Syntaxvgm where can i get that benchmark i cant find it anywhere

I can give it too you when i get home. I just left for a car repair. Nvidia has been dmca attacking the links

[hiyayhi]

Super interesting post.

 

I am honestly surprised (Once again, faith in Nvidia getting lower), that Nvidia didn't themselves notice this when all of this info was coming to light, and tell everyone.

 

 Seriously Nvidia, get your shit together. Once you realize you dun fucked up in marketing your new GPU, why would you not do thorough tests using what other people have used to show you. I know they knew what the problem was, but seriously, they should still have been scrutinizing the tests and looking into it further than what they did. (That's just what I can see from this, I could be completely wrong and they did realise just not publicly.)

[Zweihander]

Dude, your sig.... I am successful, therefore too much time on my hands.  B)

[SuperPug]

I can give it too you when i get home. I just left for a car repair. Nvidia has been dmca attacking the links

 

I found the original post by the Dev (Nai) on a german forum. And i justrealized that the benchmark only works on CUDA and doesnt work on AMD cards

 

Here's the download for anyone interested: http://www.computerbase.de/forum/showthread.php?t=1440618

[Syntaxvgm]

I found the original post by the Dev (Nai) on a german forum. And i justrealized that the benchmark only works on CUDA and doesnt work on AMD cards

Here's the download for anyone interested: http://www.computerbase.de/forum/showthread.php?t=1440618

After reading some of Nais German forum posts and his comments about the potential issues with his tool is what got me to actually download it and take a look myself.

[SuperPug]

Rewriting the benchmark for AMD might be pretty difficult according to Nai.

 

Here's his reasoning from the thread (rough translation for non german speakers):

 

With CUDA the data of a running process cannot displace itself from the GPU DRAM. Its only possible for data from different processes to displace each other from the GPU DRAM (hence why you're supposed to run the whole benchmark headless).

 

With OpenGL & OpenCL the driver is allowed to move/copy the buffers and textures around within the GPU DRAM and between the GPU DRAM and CPU DRAM. Hence the GPU can change the position of the Benchmark Data within the DRAM or even swap it to the CPU DRAM during the benchmark which would falsify the results. 

 

He hasn't been able to figure out how to supress the swapping yet...

[Syntaxvgm]

Rewriting the benchmark for AMD might be pretty difficult according to Nai.

 

Here's his reasoning from the thread (rough translation for non german speakers):

 

With CUDA the data of a running process cannot displace itself from the GPU DRAM. Its only possible for data from different processes to displace each other from the GPU DRAM (hence why you're supposed to run the whole benchmark headless).

 

With OpenGL & OpenCL the driver is allowed to move/copy the buffers and textures around within the GPU DRAM and between the GPU DRAM and CPU DRAM. Hence the GPU can change the position of the Benchmark Data within the DRAM or even swap it to the CPU DRAM during the benchmark which would falsify the results. 

 

He hasn't been able to figure out how to supress the swapping yet...

my initial thought with this test is that there is swapping occurring anyway, looking at the fact that the PCIe bandwidth changes the numbers. 

[SuperPug]

my initial thought with this test is that there is swapping occurring anyway, looking at the fact that the PCIe bandwidth changes the numbers. 

 

Hmmm.. Is it possible for you to run the benchmark on PCIe 3.0 16x with your Ram clocked @ 1333Mhz 1066Mhz and 1866Mhz respectively? Curious to see if reducing the system memory bandwith has the same effect on the benchmark as reducing the PCIe bandwith.

 

Man i wish i had a 970 just so i could run these tests...

 

EDIT: Ram speed was wrong...

[Syntaxvgm]

Hmmm.. Is it possible for you to run the benchmark on PCIe 3.0 16x with your Ram clocked @ 1333Mhz and 1866Mhz respectively? Curious to see if reducing the system memory bandwith has the same effect on the benchmark as reducing the PCIe bandwith.

 

Man i wish i had a 970 just so i could run these tests...

I have a sandy system at home. I may have a friend with a 3.0 system though, and I know a couple of the systems in my uni have it. So we'll see. I already wanted to do that, but we'll see. The fact it changes this though proves something else is up and I want to take a look at the code and make my own solution if I can with the time i have. 

 

Just to add, I do have a few ideas why this is happening, and if I decide to make my own thing, if it can be proven reliable, I would like to fashion it into a permanent tool we can use in the future for this sort of thing. After this firestorm, reviewers and users will want a good test for BOTH AMD and NVIDIA cards. The AMD promise is talking out my ass for now, as while I have a few ideas to do this avoiding using Cuda, I have noy tried any of them yet or taken a hard look at the feasibility of it. 

[Dabombinable]

http://linustechtips.com/main/topic/303145-i-found-a-problem-with-the-970-memory-test-im-examining-nais-benchmark-and-possibly-making-my-own/

^^

About Nai's test

Was going to post this on your thread, but I might be having to show the results here instead:

"Looking at the results you received, I will be testing with the following PCIe specs and the GTX 970 at stock:

PCIe Gen 2 @x2

PCIe Gen 2 @x4

PCIe Gen 1.1 @ x16

PCIe Gen 2.0 @ x16

PCie Gen 3.0 @ x16

 

I noticed 2 discrepancy in the DRAM bandwidth test, and 2 in the L2 cache test."

[XTankSlayerX]

I would like to test this out as well....

 

I have a 770. Could you send me the tool as well?

 

(Not 100% sure if I know whats going on)

[MLG Doge]

-Ignore pls- 

[SuperPug]

I would like to test this out as well....

 

I have a 770. Could you send me the tool as well?

 

(Not 100% sure if I know whats going on)

 

Posted link above... Here it is again for your lazy ass http://www.computerb...d.php?t=1440618

[SuperPug]

Was going to post this on your thread, but I might be having to show the results here instead:

"Looking at the results you received, I will be testing with the following PCIe specs and the GTX 970 at stock:

PCIe Gen 2 @x2

PCIe Gen 2 @x4

PCIe Gen 1.1 @ x16

PCIe Gen 2.0 @ x16

PCie Gen 3.0 @ x16

 

I noticed 2 discrepancy in the DRAM bandwidth test, and 2 in the L2 cache test."

 

Can you run the benchmark on PCIe 3.0 16x with Ram @ 1333Mhz 1066Mhz and 1866Mhz respectively?

[Dabombinable]

Can you run the benchmark on PCIe 3.0 16x with Ram @ 1333Mhz and 1866Mhz respectively?

I have 3x 1333MHz 4GB sticks, and 1x 1600MHz 4GB. I've never seen the hype over higher speeds as when set to 800MHz, I notice no difference at all in anything I do.

[SuperPug]

I have 3x 1333MHz 4GB sticks, and 1x 1600MHz 4GB. I've never seen the hype over higher speeds as when set to 800MHz, I notice no difference at all in anything I do.

 

Correction: meant to say 1066Mhz & 1866Mhz. And its about testing the impact of increasing the system memory bandwidth on the benchmark. Running the Ram @ 1066Mhz and 1866Mhz with PCIe 3.0 16x should have a similar effect on the results as running the ram @ 1066Mhz with PCIe 2.0 16x and 8x as OP did. Thats what im curious to find out.

[Dabombinable]

Correction: meant to say 1066Mhz & 1866Mhz. And its about testing the impact of increasing the system memory bandwidth on the benchmark. Running the Ram @ 1066Mhz and 1866Mhz with PCIe 3.0 16x should have a similar effect on the results as running the ram @ 1066Mhz with PCIe 2.0 16x and 8x as OP did. Thats what im curious to find out.

800MHz-1600MHz (I can run Windows 7 off 1 stick with no problems) will give similar testing conditions, and more accurate results as I will test across 4 different speeds.

[SuperPug]

800MHz-1600MHz (I can run Windows 7 off 1 stick with no problems) will give similar testing conditions, and more accurate results as I will test across 4 different speeds.

 

I picked those speeds specifically because of their bandwidth tho:

 

DDR3-1066 = ~  8000Mb/s peak (8533)

DDR3-1866 = ~15000Mb/s peak (14933)

 

PCIe 3.0   8x = ~  8000Mb/s peak (7877)

PCIe 3.0 16x = ~15000Mb/s peak (15754)

 

I kinda wanted to see if reducing the PCIe bandwidth has the same effect as reducing the System Memory bandwidth.

 

But im sure your test will yield interesting results aswell.