AMD accuses BAPCo and Intel of cheating with Sysmark benchmarks

[Dabombinable]

Why do you hate AMD so much?

What he's saying is true though. And maybe he got burned by AMD's FX line like a lot of people.

[SurvivorNVL]

Why do you hate AMD so much?

It seems he may have been one of the many burned by the FX line and false hope / promises.  I myself used to be optimistic for AMD, but disappointment after disappointment and the fact they still won't be able to properly compete and innovate in the market even if Zen is a success, due to their debt, leaves me in a position where I favorably would prefer them to just fizzle out and be cannibalized by Intel and nVidia.  We need constant innovation, and pushes for technology and AMD, while having good ideas, under-deliver, and always push to the Open-Source community, stalling down their process and progress at times.  With the mess of Freesync monitors releasing, AMD should've negotiated even just $50 to them per monitor.  They're a company, and a lot of their choices, frankly just.. don't make sense.  At all.

[Sakkura]

Which is sort of the point.

In Cinebench we can prove Haswell is about 70%faster core vs core. In games that advantage shrinks to around 30%.

Games are were we want the cpu to perform, so why doesn't the synthetic benchmark try to replicate those types of workload??

Instead the workload is geared more towards highly optimized specialized tasks that the given consumer product likely never will be doing in day to day life.

It's funny how you complain about benchmarks of niche workloads, then say they should focus on gaming - which is a niche workload. And one that's more often GPU bottlenecked.

[Jahramika]

Source PC world: http://www.pcworld.com/article/3023373/hardware/amd-accuses-bapco-and-intel-of-cheating-with-sysmark-benchmarks.html

 

By Gordon Mah Ung Executive Editor

 

'' AMD threw out a bombshell and accused its rival Intel and BAPCo, the benchmarking consortium, of cheating. 

In a video posted Thursday on Youtube, John Hampton, director of AMD’s client computing products, went so far as to refer obliquely to the recent Volkswagen scandal, where the German car manufacturer was accused of cheating on diesel emissions tests. “The recent debacle over a major auto maker provide the perfect illustration as to why the information provided by even the most established organizations can be misleading,” Hampton said.

Intel declined to comment on AMD’s accusation, but when asked BAPCo officials said its customers trusted it.

"The reason thousands of customers trust BAPCo benchmarks is because we are an industry consortium that focuses on the performance of applications that people use on a daily basis," a spokesman for the consortium said.

Why this matters: Performance still matters to consumers and organizations. Third-party benchmarks hold heavy sway over purchasing decisions even if few understand what they measure. AMD asks reasonable questions, but the answers remain murky—even from AMD.

Robert Cardin

 

AMD makes its case

Hampton laid out AMD's case in the video. “So truth or myth: is Sysmark a reliable, objective, unbiased benchmark to use in evaluating system performance?” Hampton asked. Hampton and AMD engineering manager Tony Salinas then ran two “similar” laptops running Sysmark 2014. The Core i5 laptop scored about 987, while the AMD FX laptop scored 659.

Salinas then ran the same laptop in Futuremark’s PCMark 8 Work Accelerated workload. While the AMD FX laptop is slower, it’s only about 7 percent slower.

One final test Salinas ran was an unidentified benchmark using Microsoft Office. The Core i5 finished in 61 seconds, while the FX chip finished in 64 seconds.

 

“What we concluded is that Sysmark does not use realistic every day workloads,” Hampton said. He encouraged viewers to read the FTC’s fine print, which dictated what Intel had to disclose on benchmarks.

The FTC ruling in 2010 bound Intel to say: “Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as Sysmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchase, including the performance of that product when combined with other product.”

 

A longstanding feud

AMD’s problems actually go all the way back to 2000, when the company’s Athlon XP CPU was kicking Pentium 4 butt in Sysmark 2001. When Sysmark 2002 was released, however, the Pentium 4 was suddenly the leader. After that AMD decided to join BAPCo in an attempt to have more influence over what it tested.

The company stayed in BAPCo through 2011 when, in a much-publicized blowup, it quit and walked away, accusing the test of being cooked for Intel’s CPUs. Although they didn’t say why, Nvidia and VIA left BAPCo at the same time.

BAPCo has primarily been made up of PC OEMs, along with Intel and other companies. At one point, even Apple joined BAPCo, as well as media organizations.

Sysmark uses off-the-shelf applications such as Photoshop, Premiere, Word, and Excel. It tasks the apps with a workload and then measures only the response time to the task.

AMD’s problems haven’t always been the apps, but the workloads. When it quit in 2011, the company told me at the time that it just didn’t think Sysmark exploited the “future” of computing and didn’t test the GPU.

Unsurprisingly, five years later, AMD’s complaints are the same. In the company’s video, Hampton says: “There is an excessive amount of high CPU tasking being done (in SYSMark). That is, the benchmark is really only evaluating the CPU side of the system.”

 

Benchmarking vs. benchmarketing

Part of the problem is the politics behind benchmarking—the not-so-fine line when it might turn into "benchmarketing," when numbers and tests are cherry-picked to make one product look better than the other. In this case, AMD is likely telling the truth that BAPCo 2014 1.5 focuses mostly on pure CPU performance. But isn’t that what it’s supposed to do? Measure the CPU performance?

From AMD’s perspective, no. The company has long insisted the future is about GPU computing. And, well, no surprise, AMD has also long enjoyed a performance advantage over Intel’s CPUs in graphics performance.

In fact, one of the tests AMD uses to show it’s behind Intel, but not that far behind, is PCMark 8 Work Accelerated. The test has two options: One uses OpenCL, which taps the GPU, while the other relies on just the CPU.

This begs the question: What was the score on that same laptop if the GPU wasn’t factored into it? Is there a little benchmarketing going on there from AMD?

You’d also have to ask yourself, how many common work or office apps today heavily rely on OpenCL? Few to none, I’d guess.

 

What we run and why

As someone who has burned too many hours coaxing Sysmark to run on systems, I was glad to leave it behind. I didn’t have any proof it was cooked, but it took forever to install and forever to run. In those days, it would often bomb out, meaning you wasted yet another day.

The methodology seemed very solid, though. For example, rather than “type” a document at 1,000 wpm (which many Office suite tests did and still do), Sysmark found a way to “type” at realistic human speeds while measuring only the response time. 

But in 2016, who the hell cares? In 1997 we cared about typing in Word or viewing a PowerPoint, but today any PC with an SSD, enough RAM and a reasonably fast CPU does the job for 90 pecent of work tasks. Most of us could not tell the difference between a dual-core Core i3 or 8-core Core i7 chip (with proper RAM and SSD)  for standard Office drone tasks.

That’s why I often use PCMark 8 Conventional, which runs on just the CPU, to illustrate that it really doesn’t matter that much. Here’s the result from a stack of laptops. My real-world use of all of these laptops—from Haswell to Skylake, and from Core M to Core i7—confirms that I can’t tell the difference in Google Chrome, Outlook, and Word from Surface Book to a ZenBook. Atom though, that’s another story.

For most Office tasks, you’d be hard pressed to feel a difference between a Core i7 or Core m chip.

 

What you should pay attention to

The take away from this latest kerfuffle isn’t that benchmarks don’t matter, it’s that people—and testers—should apply and interpret them correctly. In Office, who cares if you have a Core i3 or FX CPU. In a video encode or a game though? Hell yes it matters. "

 

-----------------------------------

 

So Linus & Luke What's your intake on this matter ?

 

-M-

I cant remember what was the amount the courts awarded the last time accusations like these was awarded to AMD from Intel?

[Majestic]

This is some desperate shit right here.

[Nena Trinity]

Intel have always done these kinda things sheesh its nothing new... Pentium vs Athlon anyone?

[Majestic]

Intel have always done these kinda things sheesh its nothing new... Pentium vs Athlon anyone?

 

Yeah, lets dig up some fossils. Just to be extra polarising. Clearly, there is not enough tribalism yet on this forum.

[Nena Trinity]

Yeah, lets dig up some fossils. Just to be extra polarising. Clearly, there is not enough tribalism yet on this forum.

Integer?

[Sakkura]

Yeah, lets dig up some fossils. Just to be extra polarising. Clearly, there is not enough tribalism yet on this forum.

That's polarizing, but your post about desperation isn't?

[Majestic]

That's polarizing, but your post about desperation isn't?

 

No, because it is desperate to go for these tactics. Regardless of which brand is doing it.

[Sakkura]

No, because it is desperate to go for these tactics. Regardless of which brand is doing it.

That's your polarizing opinion. What if they're right? We know Intel has done this before.

[Jahramika]

Well you know once you steal, bear false witness or cheat to win that label stays with you or your company until people believe you again, if ever. Only get so many chances in life to make a first impression hard to overcome your past that makes you who you are. Can bring all the excuses you want facts remain they did it before Intel. With all there money they dot care because they wont have to pay for it until 10 years later.

[zMeul]

That's your polarizing opinion. What if they're right? We know Intel has done this before.

Intel did this before because their P4 was actually not that good; today, with Intel's lineup, there is absolutely no need to - they are 3+y ahead of anything offered by AMD

there are instances where a bloody i3 stomps the shit out of a FX, and not by a small margin either : http://www.anandtech.com/show/8427/amd-fx-8370e-cpu-review-vishera-95w/2

[Jahramika]

So AMD are babies even thought 10 of your fellow citizens found them guilty before? AMD who has has been abused in this way before is wrong to be concerned that it will continue now? Please don't be dumb to think that Intel will not do whatever they can to WIN at all costs a fine of a billion dollars means nothing to Intel if the kill AMD.

[VagabondWraith]

There goes AMD crying again. When they can't compete, they cry foul. Apparently, a last resort is to blame others when you only make shitty, uncompetitive products. Crying and blaming others (Intel/nVidia) won't make your products any better. As much effort they put into bitching and moaning, they could be spending time developing good products. Alas, AMD gonna AMD.

[Jahramika]

Hey cracker the past actions of Intel by a Jury says Intel plays that way. It is not crying to acknowledge the truth of the past. That is why there is  a class in school called HISTORY so we learn not to be so ignorant to repeat it unlike you who does not learn from the past.

[AresKrieger]

So AMD are babies even thought 10 of your fellow citizens found them guilty before? AMD who has has been abused in this way before is wrong to be concerned that it will continue now? Please don't be dumb to think that Intel will not do whatever they can to WIN at all costs a fine of a billion dollars means nothing to Intel if the kill AMD.

They don't want to kill amd that's not in their best interest, they wait for amd to build itself back up before beating it up that way they can avoid being dismantled

[patrickjp93]

Right, now all accusations are playing victim and trying to get attention, regardless of how many times the other party has been proven guilty of similar actions.

Oh my God, really? AMD has lied through their teeth before. I want solid proof and Richard Huddy 10 miles from the scene before I believe a word of this. BAPCo prides itself on its integrity; and for all of Intel's influence, there are (iirc) 9 other members? AMD is using Intel's guilt under Otellini against them. Kirzanich is nothing like him, but victimhood sells. Proof or go home. AMD should put its money where it's mouth is and sue. Otherwise there is 0 reason to believe them outside independent review.

[Jahramika]

One thing to stretch the truth or lie to make your own product look better than it is and not harm another's product, it is not ethical but not a crime. To lie in a attempt to harm another's product because yours does not match up is a crime and why a jury awarded Amd to receive money from Intel in the past. So don't use a crap argument that they or Amd lies. Amd never had to pay Intel for harming their product.

[Jahramika]

Oh and if there is no past or present guilt on Intel part there would be nothing to feel guilty about. Well you are here defending them so you must think they are guilty to.

[AresKrieger]

Oh and if there is no past or present guilt on Intel part there would be nothing to feel guilty about. Well you are here defending them so you must think they are guilty to.

That argument doesn't make sense, I deny the crimes therefore I am guilty is essentially what that statement is.

[Jahramika]

Conversation isn't just about this time it's also about last time and Intel was found guilty bathe courts and made to pay. Denie all you want it's already history.

[AresKrieger]

Conversation isn't just about this time it's also about last time and Intel was found guilty bathe courts and made to pay. Denie all you want it's already history.

Ok so one company's past actions matter while the other's is irrelevant? Believe what you want, as I already stated intel doesn't currently want to destroy amd, frankly it would be fiscally damaging for them to do so, either way both companies are not saints so the best way to figure out the likely truth is to follow the money.

[PerfectTemplar]

AMD clutching to whatever they can get their hands on.

[Luke]

AMD clutching to whatever they can get their hands on.

 

Or they're just calling out Sysmark for their benchmark being bias towards intel? If that's the case then it could be problematic for their new CPU release but if not then feel free to speak up and let us know.