AMD Gives BAPCo the Bird, Chirps About Unfair SYSMark 2012 Benchmark

AMD Gives BAPCo the Bird, Chirps About Unfair SYSMark 2012 Benchmark

In a shocking move this morning, AMD bid the BAPCo (Business Applications Performance Corporation) consortium a not-so-found farewell and stated it will no longer endorse SYSmark 2012 (SM2012), a benchmark that historically has been accused of being pro-Intel. Among other things, one of AMD's chief complaints is that SM2012 virtually ignores GPU performance, a particularly egregious offense in AMD's eyes given the chip maker's infatuation with Accelerated Processing Units (APUs).

"SM2012 doesn’t represent the evolution of computer processing and how that evolution is influencing average users’ experience," AMD's Nigel Dessau wrote in a blog post. "SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today’s business environments."


AMD has long been a member of BAPCo, an organization whose goal is to create meaningful benchmarks in an applications environment. But it's a goal AMD felt BAPCo was failing, in particular with SYSmark, and the chip maker says it tried in vain to get the next-generation SM2012 benchmark right. What's more, BAPCo's failure to turn SYSmark into what AMD feels would be a meaningful benchmark could cost the industry billions of dollars, AMD claims.

"The heart of our complaint is this: the SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor’s designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8B!," Dessau estimates.


As bold a move as this is for AMD, the chip maker might not be the only one turning its nose up at SM2012 and resigning from BAPCo. Semi Accurate reports that Nvidia and VIA are pulling out too, though neither company is speaking out like AMD has chosen to do. Remaining members include ARCIntuition, Dell, Hewlett-Packard, Hitachi, Intel, Lenovo, Microsoft, Samsung, Seagate, Sony, Toshiba and others.

AMD brings up an interesting discussion point in terms of benchmarking, one we'd like our readers to chime in on. The question is this: Do you think the computing paradigm is shifting towards GPU performance, and if so, should benchmarks like SYSmark put a bigger focus on GPU performance, or do you think it's still too early in the game for that?
0
+ -

I think SYSmark could address these concerns by marketing different benchmarks that test either CPU performance, GPU performance, or both. Also while GPU performance is becoming more important I thought the CPU was still the deal breaker for heavy duty computing like rendering and re-encoding.

0
+ -

I absolutely agree with AMD on this. This is all part of the Fusion APU.

GPU is more important today than ever, hence Intel merging the CPU with the GPU and others following suit.

AMD feels these benchmarks are skewed and IMO they are correct.

0
+ -

i feel that it is shifting towards gpu based processing, however- why don't they disclose what sysmark actually tests, CPU?

similar thought to Lbowen

0
+ -

I kinda feel like the GPU is important in terms of everyday experience, but not necessarily heavy duty computations. Certainly thats changing with microsofts C++ initiative and what ive read about AMD's next gen GPU. So i guess what im getting at is that the change should be more incremental as the GPU is slowly integrated into heavier computing duties.

0
+ -

The GPU is rapidly becoming part of what is traditionally referred to as a CPU. I get the impression that sysmark is falling behind if they don't include some type of integrated GPGPU benchmark with their CPU benchmark suite. Intel and AMD are both pushing for the technology to move forward in regards to the integrated GPU on Die or APU as AMD puts it. Not to mention that the latest models already have integrated GPUs on the die.

Especially after Microsoft just made that huge announcement about C++ working off of any directX GPU. I for one am really looking forward to running multithreaded applications much faster and more efficently than ever and that sysmark should be preparing for the near future

0
+ -

I am with AMD on this one as well, and was just thinking of the remarks from the latest HH vidcast something like "We are a just a generation away from some really amazing things " having the gpu on die is certainly one of these.I think that it really shifting towards more gpu intensive because for me it's what is most noticeable with the computing experience these days.

+1
+ -

I don't really know enough of the technical details to comment on this, except where it concerns my own experiences.

An Example:

I bought a (cheap,.....and I mean really, really cheap) N68 Chipset based Motherboard and a Phenom-II 3.0 GHz. dual core CPU. It has DDR2-800 RAM (2GB) in it and its performance lends credence to the old saying, 'you get what you pay for'.

It works, and it works reliably, but it's nothing special as far as its performance goes.

Then I noticed that I had a XFX Radeon HD6870 Black Edition GPU sitting on the shelf, doing nothing at all.

In the box,.....collecting dust.

I wondered what difference it would make in this little Phenom system, and installed it right away.

Holy Crap!

It was like Popeye and his can of Spinach! All of a sudden that little 'puter was a gaming machine. Maybe not the same as my two i7-870's are, but it's hooked to a 27" ASUS Screen and I just left it there. I have been using it off and on ever since. Yeah, I left it connected, and I still use it.

So I can see a modicum of truth in AMD's statements. The GPU lends a lot to a system these days. Any benchmark that ignores that is selling us short by not telling the whole picture. Of course, the same goes for my GTX-570 & GTX-560Ti cards too.

As for me, I've realized that I can buy an AMD GPU and know beforehand that it will be capable of all the performance I'm willing to pay for. Since they're the underdog, I try to steer as much business their way as I can.

No matter what, I'll support them when I can.

0
+ -

The one thing I really see in our very near if not current future is the rate of speed the computer market is changing. The reason I say this is because if you think about 5 years ago, and the machine a general consumer would have compared to anything on this APU platform AMD or Intel is throwing out is significantly beyond everything in the general market in that short time span.

On top of that you look at the planned development paths and realize in less than a year anyone who buys a PC will have this hardware capability readily available at a decent price it will all speed up even further. One thing I have mentioned quite a few times about the future as I see it with general users having a home server, and many other devices around that as a home network for every member of any house, and for everything be it social, shopping, movies, television, cell phones etc etc, will be here in a very short amount of time.

Heck we will have quad core smart phones and slates in 6 months or less roughly. throw one of these in a slate and you have a quad core with a mid high Radeon 6000 video core and I think it was 48 GPU cores as well. That is in a 10"-7" gadget that costs 600 bucks or less, and 2-4 Gb of memory if not more that a 3 year old could carry rather easily.

0
+ -

Oh and as far as Sysmark and or pretty much any benchmark, they all seem to have some primary compatibility specializations. I am sure Intel, AMD, Nvidia, and any other hardware manufacturer on the larger side of things pays all of these major benchmark software building companies nice amounts of money quarterly.

0
+ -

I agree with AMD skewed tests are not valid. SYSmark should be able to have a CPU, CPU/GPU, and GPU benchmarks in order to truly show the differences in processors. Glad to see them standing up for themselves on this front to some degree.

0
+ -

I agree with AMD, provided they did actually put the amount of effort into getting Sysmark to change that they said they have (and why wouldn't they have, they need good benchmarks too.) I wonder if there's any chance they could come up with their own benchmarking software? Might not work though, they could too easily optimize it for their own hardware and be called biased I guess.

0
+ -

Intel combined CPU and GPU earlier than AMD, but they not complaining sysmark for the score they have. As far as i know, Intel GPU is not something you can't be proud of. But they have a good score in sysmark. AMD has faster onboard GPU and we already know about that. AMD doesnt need Sysmark to proclaim that AMD has better GPU onboard, that is very childish i think. Sysmark only bench CPU, so if AMD want a good score on software bench based on their onboard GPU;s ... find other software house that can make it happen. Walk away from Bapco clearly ensure their defeat agains Intel CPU's and pretend to be not. that's my oppinion on this situation. Like my old man said : Loser always complain.

Login or Register to Comment
Post a Comment
Username:   Password: