Intel wants reviewers to benchmark using windows media player instead of cinebench for low end mobile

[spartaman64]

Quote

Since at least Computex, Intel has been raising concerns with reviewers about the types of tests we run, which applications reviewers tend to use, and whether those tests are capturing ‘real-world’ performance. Specifically, Intel feels that far too much emphasis is put on tests like Cinebench, while the applications that people actually use are practically ignored.

Quote

Every company has benchmarks that it prefers and benchmarks that it dislikes. The fact that some tests run better on AMD versus Intel, or on Nvidia versus AMD, is not, in and of itself, evidence that the benchmark has been deliberately designed to favor one company or the other. Companies tend to raise concerns about which benchmarks reviewers are using when they are facing increased competitive pressure in the market. Those of you who think Intel is raising questions about the tests we reviewers collectively use partly because it’s losing in a lot of those tests are not wrong. But just because a company has self-interested reasons to be raising questions doesn’t automatically mean that the company is wrong, either. And since I don’t spend dozens of hours and occasional all-nighters testing hardware to give people a false idea of how it will perform, I’m always willing to revisit my own conclusions.

Quote

Intel Proposes an Alternate Approach
Up until now, we’ve talked strictly about whether a test is real-world in light of whether the results generalize to other applications. There is, however, another way to frame the topic. Intel surveyed users to see which applications they actually used, then presented us with that data. It looks like this:

https://imgur.com/a/fA17948

Quote

The implication here is that by testing the most common applications installed on people’s hardware, we can capture a better, more representative use-case. This feels intuitively true — but the reality is more complicated.

Just because an application is frequently used doesn’t make it an objectively good benchmark. Some applications are not particularly demanding. While there are absolutely scenarios in which measuring Chrome performance could be important, like the low-end notebook space, good reviews of these products already include these types of tests. In the high-end enthusiast context, Chrome is unlikely to be a taxing application. Are there test scenarios that can make it taxing? Yes. But those scenarios don’t reflect the way the application is most commonly used.

Quote

I do not pick review benchmarks solely because the application is popular, though popularity may figure into the final decision. The goal, in a general review, is to pick tests that will generalize well to other applications. The fact that a person has Steam or Battle.net installed tells me nothing. Is that person playing Overwatch or WoW Classic? Are they playing Minecraft or No Man’s Sky? Do they choose MMORPGs or FPS-type games, or are they just stalled out in Goat Simulator 2017? Are they actually playing any games at all? I can’t know without more data.

The applications on this list that show meaningful performance differences in common tasks are typically tested already. Publications like Puget Systems regularly publish performance comparisons in the Adobe suite. In some cases, the reason applications aren’t tested more often is that there have been longstanding concerns about the reliability and accuracy of the benchmark suite that most commonly includes them.

Quote

I’m always interested in better methods of measuring PC performance. Intel absolutely has a part to play in that process — the company has been helpful on many occasions when it comes to finding ways to highlight new features or troubleshoot issues. But the only way to find meaningful differences in hardware is to find meaningful differences in tests. Again, generally speaking, you’ll see reviewers check laptops for gaps in battery life and power consumption as well as performance. In GPUs, we look for differences in frame time and framerate. Because none of us can run every workload, we look for applications with generalizable results. At ET, I run multiple rendering applications specifically to ensure we aren’t favoring any single vendor or solution. That’s why I test Cinebench, Blender, Maxwell Render, and Corona Render. When it comes to media encoding, Handbrake is virtually everyone’s go-to solution — but we check in both H.264 and H.265 to ensure we capture multiple test scenarios. When tests prove to be inaccurate or insufficient to capture the data I need, I use different tests.

Quote

The much-argued difference between “synthetic” and “real-world” benchmarks is a poor framing of the issue. What matters, in the end, is whether the benchmark data presented by the reviewer collectively offers an accurate view of expected device performance. As Rob Williams details at Techgage, Intel has been only too happy to use Maxon’s Cinebench as a benchmark at times when its own CPU cores were dominating performance.

Quote

Update (9/5/2019): One thing I didn’t call out is the fact that Intel’s data set of most commonly used applications is drawn entirely from notebook and 2-in-1 devices. It discloses this in the slide above. We would not expect that content creators working in 3D applications like Blender, Cinebench, or similar workstation-class applications would be using 2-in-1’s. The implication that these applications are less important because of a low installation base is undercut by the fact that the hardware configurations Intel measured are not representative of the systems where we would expect these applications to be used.

source: https://www.extremetech.com/computing/297864-intel-is-suddenly-very-concerned-with-real-world-benchmarking

 

I think intel does have a point in that some synthetic tests are not all that useful other than giving a general idea. But i think its clear to pretty much everyone that intel only has a problem with it now that they are losing in them. Also of the applications that intel lists only 2 of them (csgo and LoL) would be useful at all to benchmark for modern desktop processors as I don't think whether or not being able to run chrome is much of a concern. And I'm not sure how you would even benchmark some of them like one drive or the steam/battlenet client. And of course listening to intel about what to benchmark and present to consumers seems like a big conflict of interest. Also I think the author of the article is trying to say that the line between synthetic and real world benchmark isn't that clear cut. For example some games have a built in benchmark tool would that be considered synthetic? 

 

intel responded

Quote

I have also seen some chatter about other data points and messages from our IFA event. Something that I simply didn’t think about before the slides for the event were posted publicly was how much of the discussion, context, and transitions between topics were handled through voice over. Press that weren’t in attendance (or the enthusiast community that looked at the slides separately) didn’t have that advantage, and as such, some of the ideas could be misconstrued or taken out of context.
For example, back to back slides go from our performance per dollar preview of the upcoming Cascade Lake-X processors immediately to a slide showing “82%” with an image of Cinebench. Without context, I can easily see how you assume we are talking about the same product families and topics, but in reality, this was when I moved from our short desktop introduction section to talk about mobile.

Quote

Finally, I wanted to apologize to Roman Hartung (Der8auer) as well as the authors quoted from Guru3D and PC Perspective for not personally giving them a heads up on our plan to use their data in our presentation.
I hope this helps to settle some of the questions and concerns floating around the hardware community about our presentation deck and the messaging supplied by them. It’s clear that the context of these presentations is critical to properly explaining our point of view, so I will definitely be paying attention to that going forward. I welcome any other questions or feedback and I’ll try to respond to as many as I can. Happy Friday!

https://medium.com/performance-at-intel/food-for-thought-on-our-ifa-event-395dc775de9

ok it seems like the presenter just didnt think about how the slides will be interpreted for people who arent at his presentation. i know ryan shrout from his pc per days and if he was the one who made the slides i trust him enough to give him the benefit of the doubt and say thats not his intention to mislead people. but i still dont think windows media player is a good benchmarker  

[Sakkura]

Chrome? Okay maybe Cinebench isn't great, but Chrome?

[For Science!]

I'm more concerned about the 0.22% running cinema4D on their tablets.....

[5x5]

Intel trying to play dirty again by asking tech outlets to skew things in their favour. I thought that's what CTS Labs was for.

[5x5]

Also, deb8uer is not happy that Intel are using his videos in a misleading way.

 

[spartaman64]

32 minutes ago, Sakkura said:

Chrome? Okay maybe Cinebench isn't great, but Chrome?

but what about one drive or windows media player? 

[mr moose]

I'm not sure i understand the hate here,  If we look at the development difference between Intel and AMD over the last decade,  AMD have been very multi core and multi thread process heavy in their development aiming for the power user.  Intel have been directly improving single thread with more focus on HT than more cores.  It is obvious Intel have been aiming at providing for the majority of consumers and thus don't appreciate that those efforts are not reflected in most bench marking. 

 

I think had the CPU been on the other foot and benchmarks were ignoring the areas where AMD excels and they wanted a more broad review process it would be a very fair request.  

 

But lets be honest, this only applies to production workloads and general office work, there is no other way to accurately benchmark a CPU in gaming situations and Intel are already on top there.

[pas008]

58 minutes ago, spartaman64 said:

 

source: https://www.extremetech.com/computing/297864-intel-is-suddenly-very-concerned-with-real-world-benchmarking

 

I think intel does have a point in that some synthetic tests are not all that useful other than giving a general idea. But i think its clear to pretty much everyone that intel only has a problem with it now that they are losing in them. Also of the applications that intel lists only 2 of them (csgo and LoL) would be useful at all to benchmark for modern desktop processors as I don't think whether or not being able to run chrome is much of a concern. And of course listening to intel about what to benchmark and present to consumers seems like a big conflict of interest. Also I think the author of the article is trying to say that the line between synthetic and real world benchmark isn't that clear cut. For example some games have a built in benchmark tool would that be considered synthetic? 

they are only losing in multithreading ones (according to tpu 3700x review performance summary  its losing alot to intels 6 cores for gaming except in cpu tests where it wins by 10ish percent) 

 

but a real world like benchmark would be nice

I remember back in sandy bridge/fx days there were a couple reviews on multitasking

was nice because many assumed fx multithreading score showed better then intels 2500k but lost in multi tasking in reviews both were different tests on their own which was even nicer

but almost everyone multitasks alot while gaming some dont really hurt at all but keep chipping small things away does affect things

browser with tabs open or video going

maybe chat program going

prolly steam/uplay/epic/etc going

maybe monitoring software

etc

etc

then actually gaming or doing some productive/work/hobby shit

 

itd be nice to see how these things match up then with closer look

[Zando_]

12 minutes ago, spartaman64 said:

but what about one drive or windows media player? 

Excel benchmarks when, that's what I work in most days 

[Guest]

Just now, Zando Bob said:

Excel benchmarks when, that's what I work in most days 

Since we still can't post images.... https://www.techspot.com/review/1678-amd-ryzen-threadripper-2990wx-2950x/page2.html

2nd chart.

[spartaman64]

11 minutes ago, mr moose said:

I'm not sure i understand the hate here,  If we look at the development difference between Intel and AMD over the last decade,  AMD have been very multi core and multi thread process heavy in their development aiming for the power user.  Intel have been directly improving single thread with more focus on HT than more cores.  It is obvious Intel have been aiming at providing for the majority of consumers and thus don't appreciate that those efforts are not reflected in most bench marking. 

 

I think had the CPU been on the other foot and benchmarks were ignoring the areas where AMD excels and they wanted a more broad review process it would be a very fair request.  

 

But lets be honest, this only applies to production workloads and general office work, there is no other way to accurately benchmark a CPU in gaming situations and Intel are already on top there.

yep i agree with the general idea but the applications intel is suggesting are ridiculous. 

[Thaldor]

I can see that Intel has a point, but they also miss a huge point: The synthetic tests are there because they stress any CPU to the max and give some number to every. Do they reflect perfectly normal usecase? No. Are they biased? Maybe. But for real, what is the point to test something like how fast Chrome starts or loads a page? Like even with mid-range CPUs we start to have time differences in nanoseconds and at that point the difference is so small that even one bit in a different place in RAM starts to affect the results. Not to even talk about that those applications Intel wants people to benchmark might actually give even worse results with the top-of-the-line CPUs just because they are nowhere near optimized to take any kind of advantage from them (because there is no reason to use time to do something that stupid because no one in real use would notice any difference running them on i7-7700K or on i9-9980XE).

And about LoL and CSGO. Does Intel really have some kind of kink to be humiliated even more? Like those games are the point of the golden needle that's 100m tall standing in a square kilometer empty parking lot why you should never use most used applications as benchmarks. As vanilla neither one of those takes any toll on any CPU greater than a potato and if you remove every single limitation there is to get them stress CPUs as much as possible even the most recent mid-tier CPUs run the games "fast" enough to start breaking the game engine. Like one example test on CSGO I found (sorry, in Finnish, but you get the point, low-settings 1080p 2xMSAA, RTX 2080 Ti), like "great, would really pay the extra bucks for i9-9900K to play CSGO 11FPS slower than with i7-9700K".

[Zando_]

3 minutes ago, Den-Fi said:

Since we still can't post images.... https://www.techspot.com/review/1678-amd-ryzen-threadripper-2990wx-2950x/page2.html

2nd chart.

 Time to tell the boss I need a custom PC with a 7980XE because I totally need one for 90% VLOOKUPs. 

[mr moose]

2 minutes ago, Den-Fi said:

Since we still can't post images.... https://www.techspot.com/review/1678-amd-ryzen-threadripper-2990wx-2950x/page2.html

2nd chart.

That first graph is funny, it kinda raises more questions like: Why would you test TR or i7's for office applications?  who is going to spend that much on an office machine?  most people would be looking at the mid to bottom end.

[Dabombinable]

Ummm...you have to go back to the days of the Pentium III to be really able to do any benchmarking in WMP...are their new CPU going to be that shit compared to Ryzen 2?

[leadeater]

11 minutes ago, mr moose said:

Intel have been directly improving single thread with more focus on HT than more cores.  It is obvious Intel have been aiming at providing for the majority of consumers and thus don't appreciate that those efforts are not reflected in most bench marking.

Problem is Intel have been removing HT from a lot of their products, and AMD SMT performance scales slightly better. Agree with everything and the main point, just annoying that Intel disabled HT so readily on so many of their products. Have to segment products somehow though.

[Drak3]

Just now, leadeater said:

Have to segment products somehow though.

Easy

C/T

8/16

6/12

4/8

4/4

[mr moose]

2 minutes ago, leadeater said:

Problem is Intel have been removing HT from a lot of their products, and AMD SMT performance scales slightly better. Agree with everything and the main point, just annoying that Intel disabled HT so readily on so many of their products. Have to segment products somehow though.

I think they do that so they can still shift product in a market that doesn't need HT (just like they still have dual core CPU's because the market still has a place for them with people who don't need or want to pay for anything bigger).  I think if they don't disable HT and charge a little less they would lose sales to either AMD or their lower tier products.

[Sauron]

Yeah, let me just buy a 500$ cpu to run chrome

[pas008]

9 minutes ago, leadeater said:

Problem is Intel have been removing HT from a lot of their products, and AMD SMT performance scales slightly better. Agree with everything and the main point, just annoying that Intel disabled HT so readily on so many of their products. Have to segment products somehow though.

you have reliable article/link on them having to remove/disable ht on many products

havent seen anything regarding this recently was thinking about this awhile ago then thought every probably quietly swept it under a rug or wasnt a problem in their eyes

[The King of the Undead]

I am sorry what?!!

[79wjd]

23 minutes ago, mr moose said:

That first graph is funny, it kinda raises more questions like: Why would you test TR or i7's for office applications?  who is going to spend that much on an office machine?  most people would be looking at the mid to bottom end.

You'd be amazed how bad some IT departments are at identifying/getting the correct/reasonable hardware.

 

Whether it be the fault of the IT department being incompetent or whoever places the order not listening to the IT department's recommendation.

[Zando_]

8 minutes ago, Sauron said:

Yeah, let me just buy a 500$ cpu to run chrome

Maybe Chrome is the reason they're pushing hexa-channel and soon octa-channel RAM? Maybe we've looked at it all wrong for years, and Intel has only ever been trying to create the best CPU and platform for Chrome.... 

[The King of the Undead]

What if we use FF?

[leadeater]

8 minutes ago, pas008 said:

you have reliable article/link on them having to remove/disable ht on many products

havent seen anything regarding this recently was thinking about this awhile ago then thought every probably quietly swept it under a rug or wasnt a problem in their eyes

You're talking about the security concerns correct? Those are all fixable. Intel disabling HT on i5's etc is product segmentation so create cheaper products.