Start Using a C(G)P(U)I?

[Ryoutarou97]

 For those who are unaware, the Consumer Price Index (CPI for the sake of the rest of this post, I'm not referring to the mouse measurement CPI) is essentially a basket of goods that economists sum up (there's some weighting there, but you get the point) the prices of and then compare to last year. By using a variety of goods they can avoid shocks (say a report on why blueberries cause cancer makes sales drop 75% forcing the price way down. If they were the only thing measured then people would think there was -75% inflation) screwing with the data. Then they compare to the price from the last year's CPI to see inflation and a bunch of other stuff, but that's about as much explanation is needed for this.

 

In GPU videos, Linus and Luke use mostly the same games at the same settings and the same runs for benchmarking. Why not make an index for measuring GPUs against each other too? Since GOUs can flip flop ranking based on the game, why not sample 10 or 20 games as one would for a benchmark then compound the data? For example, an average (or sum, I can't decide which is better) of the agv. FPS in every game in the index could be calculated giving something like this (using average. Keep in mind that these numbers are just estimations and not good ones):

 

GPU Index (not actually since I don't have most of these games, this is just an example) calculated by:

 

[(BF4 Avg FPS) + (Crysis 3 avg. FPS) + (Bioshock Infinite) + (the same thing for 13 more games)] / 16

 

980:   78

970:   69

290x:  67

290:   63

280x:  58

285:   54

960:   53

 

Possibly it could be done on 3-5 CPUs or testbenches for more stability (of measurement), but if it's done on a totally overkill system like most benchmarks are, it would be a good measure of GPU power. Basically it's like using a gaming benchmark with the consistency of a synthetic. I'm not saying to end the old way, but maybe just add this as an extra graph after the usual benchmarks.

 

 

Thoughts?

[Prastupok]

 snip

 

Why not?

 

Because you can't overclock bread.

 

CPI is not equivalent at all to GPU/CPUs, as they vary.

[Samfisher]

snip

 

Because average FPS as it is is already a terrible way of measuring performance.  Having a number as an average of 20 averages make it worse as it tells the user nothing about how it actually performs..

[Ryoutarou97]

Because average FPS as it is is already a terrible way of measuring performance.  Having a number as an average of 20 averages make it worse as it tells the user nothing about how it actually performs.

What's wrong with measuring average FPS? Is it something with frame time, because that could be weighted and added to the 16 in the denominator. It shouldn't be too much of a problem since I'm pretty sure all the data gets stuffed into a spreadsheet and an excel function for it would only have to be written once. Really, this wouldn't take too much work (unless adding a graph when editing a video does. I'm not a video editor so I wouldn't know.)

[StainlessSR]

Here would be a problem with that

 

in order for the index to be comparable across all gpu's then they would all have to play the games being tested at the same settings. This would cause some gpu's to "hit a wall of fps"(ie the high end one's) or they would slide show (low end gpu's). The metric would only be valid for the cards that the settings were optimized for (mid range cards if you were using mid range settings). Then add in games that are optimized for a "Brand" of gpu's (nvidia is famous for this, and AMD is catching up) further muddying the waters. Sadly, the best way to get a ranking is still subjective (Tom's GPU hierarchy chart) rather than an absolute number. And the best way to pick a card is to match it to the games you play the most at the settings you want to play it at.
 

[Ryoutarou97]

Why not?

 

Because you can't overclock bread.

 

CPI is not equivalent at all to GPU/CPUs, as they vary.

 

It would just be a compilation of all the data normally recorded, so I don't really see the problem. By the way, I got my toaster up to 3 ghz. No more breakfast, but I do have bragging rights. Maybe I'll try liquid cooling it sometime...

[Ryoutarou97]

Here would be a problem with that

 

in order for the index to be comparable across all gpu's then they would all have to play the games being tested at the same settings. This would cause some gpu's to "hit a wall of fps"(ie the high end one's) or they would slide show (low end gpu's). The metric would only be valid for the cards that the settings were optimized for (mid range cards if you were using mid range settings). Then add in games that are optimized for a "Brand" of gpu's (nvidia is famous for this, and AMD is catching up) further muddying the waters. Sadly, the best way to get a ranking is still subjective (Tom's GPU hierarchy chart) rather than an absolute number. And the best way to pick a card is to match it to the games you play the most at the settings you want to play it at.

 

 

It would have to be sufficiently high that no current GPU could max it out, so 1440p high settings I suppose. Nothing wrong with slides showing, that just shows that it's a low end card. Using a wide array of games is to account for brand differences since it should have both brands' games mixed in. 

[Samfisher]

What's wrong with measuring average FPS? Is it something with frame time, because that could be weighted and added to the 16 in the denominator. It shouldn't be too much of a problem since I'm pretty sure all the data gets stuffed into a spreadsheet and an excel function for it would only have to be written once. Really, this wouldn't take too much work (unless adding a graph when editing a video does. I'm not a video editor so I wouldn't know.)

 

Because average FPS can mean so many things.  60FPS average could mean min and max FPS never budges from 60FPS, or it could mean 5 frames at 120fps and 5 frames at 0fps and the average is 60fps.

[blackadder]

Are you a statistician. Cause i like the way you think. I think it comes down to time. I for one would like 2000 samples or more just to feel confident abot the MOE. I would like to see a avg, well, i would love to see all the data. With game by game we get to see more details of each game, they're not lost in the avg.

[Ryoutarou97]

Because average FPS can mean so many things.  60FPS average could mean min and max FPS never budges from 60FPS, or it could mean 5 frames at 120fps and 5 frames at 0fps and the average is 60fps.

Isn't the average used in benchmarks just frames/time FPS? I doubt that FPS ever stays a a constant 60.

 

 

Are you a statistician. Cause i like the way you think. I think it comes down to time. I for one would like 2000 samples or more just to feel confident abot the MOE. I would like to see a avg, well, i would love to see all the data. With game by game we get to see more details of each game, they're not lost in the avg.

Just a student who has way too much time to prepare for the Macroeconomics AP test, but thanks  . I've not taken stats yet, so forgive me if I totally misunderstand, but isn't 2000 samples about 20-40 seconds at 30-90 FPS? I was thinking in addition to game by game it could be added at the end so you could still see game by game, but would also have a condensed form of the data to compare with other cards more quickly.

[blackadder]

Isn't the average used in benchmarks just frames/time FPS? I doubt that FPS ever stays a a constant 60.

 

 

Just a student who has way too much time to prepare for the Macroeconomics AP test, but thanks  . I've not taken stats yet, so forgive me if I totally misunderstand, but isn't 2000 samples about 20-40 seconds at 30-90 FPS? I was thinking in addition to game by game it could be added at the end so you could still see game by game, but would also have a condensed form of the data to compare with other cards more quickly.

o, its a fun class when you take it, looks like you wil enjoy it. you can make stats say anything you want, i mean anything. the 2000 samples depends of how long each sample is, i understand where you are coming from but i was making a avg and then avg them in.  if we were to break down each piece we would really get a lot of info, http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics, even the residuals is important. 

[Ryoutarou97]

o, its a fun class when you take it, looks like you wil enjoy it. you can make stats say anything you want, i mean anything. the 2000 samples depends of how long each sample is, i understand where you are coming from but i was making a avg and then avg them in.  if we were to break down each piece we would really get a lot of info, http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics, even the residuals is important. 

Sounds fun! I do like the idea of statistics, but it doesn't seem like the kind of thing that would be too easy to learn on my own, so I'll wait for the class. So, would the... yet unnamed thing I suggested be easily manipulatable to make one company look better? If so, how would one go about that?

[blackadder]

Sounds fun! I do like the idea of statistics, but it doesn't seem like the kind of thing that would be too easy to learn on my own, so I'll wait for the class. So, would the... yet unnamed thing I suggested be easily manipulatable to make one company look better? If so, how would one go about that?

say you avg more games that use nvidia game works, it it works lol. Say these games give nvidia cards a slight boost, something small even a few fps, people will see its up top and assume its better overall, this can even be a 100% true statement if your sample includes more of these games. This is called a sampling bias. Your numbers were inherently untrue to the real world population . If there are more games in the real world that use this feature and are included then this becomes a true statement in all cases. You can also do the classic messing around with the axis to make skewed lines. The list goes on. I'm not the greatest at stats too. Most of what i know comes from physics. I didn't even think stats was a real mathematics field, it is though. Has a very rich history and basis. None of the fun stuff in math is really apparent until higher up it seems, lots of theory goes into this stuff. Ill link a cool video on stats that you wouldn't think had anything to do with it in a second.