Relevant information on FPS benchmarks

[Sakerdot]

Hello, with the launch of the new Ryzen 5 I've seen that many youtubers have started adding more relevant information on FPS benchmarks, mainly some low percentiles to show the difference between the average FPS and the lowest values recorded. This is much better than just giving the highest, lowest and average values since it paints a better picture on the stability of the system. However, it requires the viewer to know a bit about math and I believe that you guys could interpret the information for the viewer a bit better than you currently are.

 

As an example I have taken some values from a recent but lightweight game, an old game with v-sync on set at 75 FPS, and two theoretical best and worst case scenarios. All of the data used will be attached to this post.

 

1. Gwent - I played a round of the game and here's the graph:

As expected it's pretty stable, those dips are only while loading and the experience is buttery smooth, here's some information about it:

Average FPS: 70.87758

Standard Deviation: 6.564846

Coefficient of Variation: 0.09262232

IQR: 7

First Quartile (25th percentile): 67.9

Second Quartile (50th percentile/Median): 74.25

Third Quartile (75th percentile): 74.9

 

2. I played on an old game for a bit, it stayed stable for basically the whole time:

And some info again:

Average FPS: 74.25301

Standard Deviation: 3.286588

Coefficient of Variation: 0.04426201

IQR: 0.075

First Quartile (25th percentile): 74.825

Second Quartile (50th percentile/Median): 74.9

Third Quartile (75th percentile): 74.9

 

3. Best case scenario:

 

Average FPS: 74.99219

Standard Deviation: 0.2346414

Coefficient of Variation: 0.003128878

IQR: 0

First Quartile (25th percentile): 75

Second Quartile (50th percentile/Median): 75

Third Quartile (75th percentile): 75

 

4. Worst case scenario, data generated with a Normal distribution of average 60 and standard deviation 20:

 

Average FPS: 60.1634428984474

Standard Deviation: 20.4591800756094

Coefficient of Variation: 0.3400599947403

IQR: 26.3127860849673

First Quartile (25th percentile): 48.1543761931225

Second Quartile (50th percentile/Median): 58.5560868919468

Third Quartile (75th percentile): 74.4671622780898

 

Now I'll explain what all of this means, I think we all know what the average means, the sum of all points over how much data there is, the standard deviation is how much, on average, a point deviates from the average (so the difference between the a point and the avg.), the coefficient of variation is the standard deviation over the average, the quartiles are the points where x% of the data is left behind, so 25% of the data remains behind the first quartile, 50% behind the second one (this is also known as the median) and so on, and the IQR is the difference between the third and first quartiles.

 

The most important piece of information here, in my opinion, is the coefficient of variation. This is because, as you can see, all of these cases have an average FPS above 60 FPS, but the first 3 ones are much more stable than the fourth one, and the only way to measure this is to take into account both the average and the standard deviation of the data, which the coefficient of variation does, the closer it is to 0 the more stable a system will be and the further appart the worse it'll be.

 

I think it'd be useful to have that coefficient given along with the average, maybe we could come up with some sort of scale based on it for those that don't know about it, what do you guys think?

Gwent FPS.txt

Old game FPS.txt

Uniform FPS.txt

Choppy FPS.txt

[Enderman]

How do you manage to get negative fps...

[vanished]

2 minutes ago, Enderman said:

How do you manage to get negative fps...

The screen captured an image and fed it back into the GPU to be converted into 3D objects

[Sakerdot]

3 minutes ago, Enderman said:

How do you manage to get negative fps...

I've explained that the last two are randomly generated, it doesn't change anything that there's a negative point somewhere.

[Enderman]

1 minute ago, Ryan_Vickers said:

The screen captured an image and fed it back into the GPU to be converted into 3D objects

Must be that new screen technology that doubles as a camera.

[vanished]

It's true that not reporting minimums is bad, and reporting minimums with not additional information isn't very useful either since that could have just been a once in a blue moon fluke, and so the correct way to do it is with percentiles.  I also think it's not that hard to understand what the numbers mean.  I think the people looking for this information already get it, and those who don't won't care.

[AshleyAshes]

Maybe the game with negative frame rate goes backwards for a brief moment?

[Sakerdot]

13 hours ago, Ryan_Vickers said:

It's true that not reporting minimums is bad, and reporting minimums with not additional information isn't very useful either since that could have just been a once in a blue moon fluke, and so the correct way to do it is with percentiles.  I also think it's not that hard to understand what the numbers mean.  I think the people looking for this information already get it, and those who don't won't care.

I agree that most people probably don't care, but data is extremely easy to manipulate, knowingly or not, and the way to avoid that is to present the most relevant information you can.

Consider the following charts:

Spoiler

 

Here we can see only averages, this is useless as we know but sometimes it is used and it can be extremely misleading. In this case the 4th system seems to be the better one.

Spoiler

 

Now we have some more information, we can expect to be 1/4 of the time below a certain value and 3/4 above it, with only this data however, you'd probably choose the 3rd system, knowing that the gap between average and 25th percentile is pretty hefty on the 4th one.

Spoiler

 

This is what I'd consider a pretty good graph, we know what to expect 1/10 and 1/4 of the time, and we can see that the 4th system is extremely unstable, the 3rd one is less unstable but still considerable if we don't have freesync or gsync, and the last 2 will give us smooth gameplay.

Spoiler

 

On this graph we can see the average along with the standard deviation, so we can expect to be most of the time on the values between those lines. With only this information we can deduce that the fourth system is very unstable, because the range of variance is huge (70-169), the third one less so (63-97) and the last two are the most stable ones. With this graph you'd probably choose between the 2nd and the 3rd one, depending on your preferences, higher fps but lower stablity or viceversa.

Spoiler

 

The most complete graph would be something like this, maybe without the 25th percentile as to not overload people with information.

Taking into account the standard deviation, without adding a lot of colors and stuff to the graphs, you can get a better representation of how the systems compare against each other. You get the most information out of it while adding the least amount of clutter, it's just a line.