Why oh why would you benchmark like that!?

[SwagLord314]

So I feel obligated to begin this by saying; I am very bias in my game selection, this influences my argument greatly. That being said this issue is a long time frustration of mine that I am curious to know if I stand alone on.

 

The bulk of the problem is with "PC Tech Reviewers". When they benchmark an item often they use a list of games that have very little correlation to (my type of) consumer interest. Let me explain. Some of the most common review benchmarks include: LOTR Shadow of Mordor, Rise of the Tomb Raider, and DOOM. At the time of writing this; not one of these were in the first 3 page folds on twitch. In fact DOOM (the first of these examples) is 85th on the list.

 

So why are they using this as an example for viewers to compare things to. I can understand the argument that so long as the tests are using the same game then you can see the difference. But I, as a consumer, don't really care how something CAN perform. I care about how it WILL perform in my use case. So to me it seems silly that no one is using Fortnite as a standard test. For better or worse Fortnite is being played more than almost any other game out there. Beyond that its free. Meaning anyone can load Fortnite up on their computer and run their own test.

 

Fortnite does not stand alone I think some other great titles could include: Overwatch, H1Z1, PUBG, Heck even Minecraft makes some sense to me!?

 

Games like this provide a much more realistic benckmark as opposed to a theoretical benchmark.

[Princess Luna]

You're basically whining for not having the games you prefer in w/e video lol

[Tabs]

Fortnite, H1Z1, Overwatch and PUBG are all inherently incapable of providing consistency in results since they are based on interaction between players or random generation. Minecraft is (heavily!) skewed towards CPU performance in it's java version and cannot be benchmarked in the bedrock version.

 

Reviewers choose the games they do because they include robust built-in benchmarking routines that provide consistency across hardware, making direct comparisons possible.

[DarkEnergy]

8 minutes ago, SwagLord314 said:

So I feel obligated to begin this by saying; I am very bias in my game selection, this influences my argument greatly.

Case closed, let's wrap it up boys.

 

While many benchmarks are chosen based on popularity, they're also chosen based on which part of the system it taxes to provide a good comparison. Also consistency. 

[SwagLord314]

4 minutes ago, Princess Cadence said:

You're basically whining for not having the games you prefer in w/e video lol

You are not wrong. However this does not speak to the argument I made for why "my" games should be used.

[Jurrunio]

15 minutes ago, SwagLord314 said:

You are not wrong. However this does not speak to the argument I made for why "my" games should be used.

Give us a way to test these 4 games in a repeatable fashion with consistent results.

[Beowulff83]

But can it run Fortnite?

[TetraSky]

They use those games for a variety of reasons.

1- Because they are easy to set up a benchmarking run that's always the same.

2- Because they are actually quite demanding on the system.

3- Because popular games on Twitch tend to be multiplayer focused games that can generally play on a potato and are nearly impossible to have the exact same benchmarking run.

Meaning if a computer can run the games in their benchmarks, they can run the current popular games on Twitch.

[dizmo]

23 minutes ago, SwagLord314 said:

You are not wrong. However this does not speak to the argument I made for why "my" games should be used.

@Tabs laid out exactly why they're poor benchmarking tools. Took the words right outta my brain.

[aezakmi]

those games are resource eaters that's why they use them.

 

Quote

Overwatch, H1Z1, PUBG, Heck even Minecraft

well those are useful to benchmark toxicity levels

[LogicalDrm]

Overwatch, Dota2, LoL and CSGO are worst games to benchmark high end gaming systems. Why? Because they are made so everyone and their mother can play those. Benchmark games are selected because of two reasons:

1) They DO stress system on higher resolution and settings

2) They have built-in benchmark function to allow consistent testing at least spent time.

[Princess Luna]

1 minute ago, LoGiCalDrm said:

CSGO

I agree a lot with CS:GO because Source 1 Engine is soooooo outdated already that it is completely derpy and inconsistent the newer the hardware, my fps on the i7 8700 is more inconsistent than it was with the i7 6700, the Ryzen 7 1800X also is all over the place... all the way from 400fps down 120fps completely randomly moving up and down regardless of the in-game settings.

[ybriK]

The answer is very simple. Fortnite is a very simple game to run. The game itself doesn't have any complex graphical/lighting effects compared to something like ROTTR or The Divison. It doesn't take expensive hardware to max out the game on a playable rate whereas ROTTR does. The whole point of benchmarking triple A demanding titles is to give you a "worst case" scenario.

 

There's a reason why some benchmarks still have Crysis 3. A game from 2013. Why? It's demanding still to this day.

[Mihle]

2 hours ago, LoGiCalDrm said:

 

2) They have built-in benchmark function to allow consistent testing at least spent time.

If nothing have changed over at LTT, they do not use the built in benchmark. I they have a spesific savepoint in the campaign that they use and try to do the same multiple times. The reason is that the built on benchmark most of the time isn't truly representative for the actual game, and might missled people to think a GPU will perform different in the game than it actually will. 

 

No YouTubers should use the built in benchmark in the games, thats just lazy if they do.

[LogicalDrm]

1 hour ago, Mihle said:

If nothing have changed over at LTT, they do not use the built in benchmark. I they have a spesific savepoint in the campaign that they use and try to do the same multiple times. The reason is that the built on benchmark most of the time isn't truly representative for the actual game, and might missled people to think a GPU will perform different in the game than it actually will. 

 

No YouTubers should use the built in benchmark in the games, thats just lazy if they do.

Linus has said in some videos that he uses built-in benchmark in Tomb Raider. They haven't talked about benchmarking procedure in a while, and I would assume they have changed stuff. Sure some games will remain as campaign runs. But as the two major testers have now different responsibilities (Luke in FPM and Jon at TQ), I think they are also using built-in benchmarks.

 

I agree that those aren't best, but neither is LMG in making reviews.

[inGameToasted]

You bring up Fortnite to benchmark, but the game is so well optimized and not demanding at all, so most of their systems would run it without any meaningful difference. That wouldn't be of any relevance to what the rigs can and can't run. Same for many of the other games you listed, except some of them like PUBG are so badly optimized that the differences wouldn't be due to the rig's specifications, but more inconsistencies in the game.

[Quadriplegic]

Because the point of benchmarking is measuring the performance of hardware in a consistent way. Twitch popularity ratings go eff themselves

[PlayStation 2]

3 hours ago, Princess Cadence said:

I agree a lot with CS:GO because Source 1 Engine is soooooo outdated already that it is completely derpy and inconsistent the newer the hardware, my fps on the i7 8700 is more inconsistent than it was with the i7 6700, the Ryzen 7 1800X also is all over the place... all the way from 400fps down 120fps completely randomly moving up and down regardless of the in-game settings.

You think CS is shit, try TF2. I guaran-fucking-tee you'll drop below 60fps.

[kokakolia]

3 hours ago, LoGiCalDrm said:

Overwatch, Dota2, LoL and CSGO are worst games to benchmark high end gaming systems. Why? Because they are made so everyone and their mother can play those. Benchmark games are selected because of two reasons:

1) They DO stress system on higher resolution and settings

2) They have built-in benchmark function to allow consistent testing at least spent time.

And this is why I don’t take these “tech review channels”. They are comparable to car salesmen who try to up sell you on features you don’t really need.

 

The fact that only the most power hungry and unoptimized games are benchmarked gives the viewer a skewed perception of the product. In effect, pc gamers tend to over build their gaming rigs because anything below a GTX 1060 won’t yield good FPS in these tough benchmark tests. 

 

This isn’t too bad because of future-proofing. Further, it is generally accepted that the cost of a decent gaming setup is usually around the $1000~$2000 range anyway. However, this just makes PC gaming seem a lot more inaccessible due to cost. If more people knew how much mileage they could get out of an old PC + GTX 1050 then there would be more PC gamers. 

 

It’s kind of the same story with other hobbies. Watch collecting is definitely the worst. If you watch any of the YouTube watch reviewers you’ll think that you’re a loser for not having a Rolex on your wrist...or a mechanical Seiko at the very least. 

 

 

 

 

[asus killer]

16 hours ago, SwagLord314 said:

 But I, as a consumer, don't really care how something CAN perform. I care about how it WILL perform in my use case. 

personalized on demand tech reviews 

[LogicalDrm]

44 minutes ago, kokakolia said:

The fact that only the most power hungry and unoptimized games are benchmarked gives the viewer a skewed perception of the product. In effect, pc gamers tend to over build their gaming rigs because anything below a GTX 1060 won’t yield good FPS in these tough benchmark tests.

I would say there's problem at both sides. Gamers/consumers are problem because they rarely do their own research. Looking at just LTT and other channels who cover actually very small part of variance with very limited testing gives that skewed view. But if you spend some time checking written reviews or smaller channels who cover more benchmarks, more variance, they give much broader look.

 

And also reviewers are issue, but as especially youtubers rely on views to keep it going, they don't want to make boring content. Like testing if there really is GPU that doesn't give 200fps on CSGO or Fortnite. Or which GPU gives 600fps on those. And since we all know that most asked question on forums is "Why isn't my beastly rig getting 600fps on CSGO?", having these benchmarks where you don't see difference doesn't make much sense either.

 

The point of benchmarks is to give some comparison between products. If you have test where results are constantly same over whole category, you quickly drop that benchmark out of loop (unless you do scientific testing). There's reason why we don't see 3DMark06 testing results anymore.

[Guest]

Benchmark games such as Doom make use of resources better and really stress the hardware. Fornite and CSGO are made to be easy to run and don’t make full use of much. 

[Mira Yurizaki]

23 hours ago, Mihle said:

If nothing have changed over at LTT, they do not use the built in benchmark. I they have a spesific savepoint in the campaign that they use and try to do the same multiple times. The reason is that the built on benchmark most of the time isn't truly representative for the actual game, and might missled people to think a GPU will perform different in the game than it actually will. 

 

No YouTubers should use the built in benchmark in the games, thats just lazy if they do.

The thing with a built-in benchmarking tool is that it's 100% (more or less) repeatable. If you do things manually, then you must be very careful to pick and choose a repeatable scenario to run every time. Otherwise you're doing something significant enough each time to make the results meaningless.

[kokakolia]

3 minutes ago, M.Yurizaki said:

The thing with a built-in benchmarking tool is that it's 100% (more or less) repeatable. If you do things manually, then you must be very careful to pick and choose a repeatable scenario to run every time. Otherwise you're doing something significant enough each time to make the results meaningless.

Meaningless results. I get that. But you’re testing for a benchmark score at the end of the day. That’s a different task than actually reviewing a product and determining its uses or potential. 

 

For example stating: “This GPU scored 55 avg FPS on Doom” isn’t as helpful as: “This GPU should run any game at 1080p on high or ultra”. 

 

Perhaps I am just bored of the act of presenting benchmarks and I just want to skip to the conclusion. 

[minibois]

From the games mentioned, they don't make a ton of sense for benchmarking due to the fact that they are multiplayer games that get constant updates or they are games that are not graphically intense that won't be interesting for (GPU) benchmarks.

 

Let me explain;

Games like Fortnite and Minecraft are made to be able to run on a wide variety of hardware, so they won't be that interesting for benchmarks. Especially Minecraft because it runs on, mainly, CPU and RAM and not much GPU. Games like Fortnite and PUBG get constant updates which tweak performance, so one weeks benchmark might not reflect next weeks performance, plus you will also be limited by factors you cannot control like how many players will be around you and your internet speed.

For example, I know in The Witcher 3 people who ran benchmarks always said "This is the fps I got in whatever town that was crowded with NPC's and this is this fps I got in a forest with no NPC, but lots of trees", but in an online game this is basically impossible to control.

 

The reason benchmarks exist is to create a point of comparison. If GPU X gets 50fps in a certain controller environment and GPU Y gets 80fps in that same controlled environment; you will know these relative differences will translate to another game too.