New Analysis Casts Doubt On Intel's Smartphone Performance vs. ARM

rated by 0 users
This post has 6 Replies | 0 Followers

Top 10 Contributor
Posts 26,703
Points 1,207,930
Joined: Sep 2007
News Posted: Thu, Jul 11 2013 3:30 PM
A few weeks ago, the analyst company ABI Research published a report claiming that Intel's new CloverTrail+ platform (dual-core Medfield) for smartphones was significantly faster and more power efficient than anything ARM's various partners were shipping. If you follow the smartphone market, that was a very surprising claim. Medfield was a decent midrange platform when it launched in 2012, but Intel made it clear that it's goal for Medfield was to compete with other platforms in its division -- not seize the performance crown outright.

The dual-core / quad-thread CloverTrail+ has improved on Medfield's performance, but it's still based on 2008-era microprocessor technology. Qualcomm, Samsung, and Apple, meanwhile, have not stood still. Last year, Cortex-A9 phones were still at the top of the market; now there are multiple devices using A15-class parts. Claiming that the Intel chip is head-and-shoulders above everything else was extremely odd.

Further investigation by other analysts has blown serious holes in the ABI Research report. Not only does it focus on a single, highly questionable benchmark (AnTuTu), the x86 version of that benchmark is running different code than the ARM flavors. Furthermore, the recently released Version 3.3 of the test is much faster on Intel hardware than on any of the other platforms.

This sort of cherry-picking and suspicious optimization makes ABI Research look foolish and raises questions about whether or not Intel has returned to its old tricks. As analyst Jim McGregor writes, "The only clear conclusions that should be drawn are that the leading ARM-based processors still have a performance lead over the latest Intel processor."

Unexpected Consequences:  SoCs Make Benchmarking More Difficult

One thing we want to highlight, however, is that benchmarking really has gotten more difficult in this brave new era of SoC products. Smartphones and tablets are essentially closed systems. Even laptops are more flexible -- a strict apples-to-apples comparison between notebooks could use identical external displays and the same RAM between multiple systems. There are also only two CPU vendors; an Intel quad-core is an Intel quad-core, regardless of whether it's in a Dell or an HP system.

SoCs are different. Every phone has a different chip and every chip is configured slightly differently, with various I/O blocks, offload processors, screen technologies, manufacturing nodes, and leakage characteristics. It's further complicated by the fact that mobile usage models are still evolving, as are the underlying technologies and capabilities of the devices themselves.  

But even with those caveats in place, the ABI Research report is bad science. Single-source performance comparisons almost inevitably are. Even an extremely comprehensive suite of tests like SiSoft Sandra is a starting point for evaluating performance, not a single end-game metric. The ABI Research team undoubtedly knew that -- which raises questions about their objectivity and intent when writing the report.

In the interest of full disclosure, ARM expressed dissatisfaction with the ABI Report when we met with them in late June and the author of the EETimes article, Jim McGregor, was also at that meeting. In this case, however, the problems with the ABI Research report are self-evident. ARM's own benchmark data comparing the Cortex-A15/A9 to Atom has to be taken with a substantial grain of salt but that doesn't change the fact that ABI is single-sourcing their conclusions from a version of a benchmark that boosts x86 performance far more than ARM.

Clover Trail+ is a good chip. We expect Bay Trail, when it launches late this year, to be a far better mobile solution. These kind of shenanigans help no one and serve only to confuse the issue.
  • | Post Points: 65
Top 10 Contributor
Posts 8,756
Points 104,950
Joined: Apr 2009
Location: Shenandoah Valley, Virginia
realneil replied on Thu, Jul 11 2013 8:27 PM

It's marketing pure and simple.

Dogs are great judges of character, and if your dog doesn't like somebody being around, you shouldn't trust them.

  • | Post Points: 20
Not Ranked
Posts 80
Points 815
Joined: Jun 2013
basroil replied on Thu, Jul 11 2013 9:36 PM

I think we can all agree it comes down to NEON vs SSE optimizations in the benchmarks. ARM will obviously ignore SSE instructions but use equivalent NEON instructions, and Intel will focus on the opposite. That said though, the fact that the Atom is always 128bit for SSE while only the A15 has 128bit NEON (A9 has 64bit at a time implementation of NEON) means performance vs A9 with full optimization on both is going to be a runaway victory for Intel.

Although the current underlying core is 64bit, this version only supports 32bit instructions, so it won't have the type of increased performance bay trail will have. Once bay trail gets out, they'll have a decent time before the A5X series gets out, during which they will entirely dominate performance.

  • | Post Points: 20
Top 150 Contributor
Posts 541
Points 4,525
Joined: Apr 2012
Location: Schertz, Texas
ajm531 replied on Thu, Jul 11 2013 11:39 PM

Well said. i mean whos gonna release not fixed benchmarks and say yeah ours isnt that great. Yeah ive seen this thru the years with android phones specifically. I remember back in the single core and even early dual core days benchmarks would get people really excited but that changed very quickly. they can be faked and mentioned in the article there are so many variables its very very hard to get accurate benchmarks. finally narrowing it down like intel did is worse publicity than showing your processor is slower.

  • | Post Points: 5
Top 150 Contributor
Posts 541
Points 4,525
Joined: Apr 2012
Location: Schertz, Texas
ajm531 replied on Thu, Jul 11 2013 11:41 PM

Sir i would like to plus 1 your comment more if i could. If i have to google half your comment you win in my book.

  • | Post Points: 5
Top 150 Contributor
Posts 619
Points 5,260
Joined: Dec 2011

Intel has been overpromising on ARM for years and that pure power means nothing we should all know by now.

  • | Post Points: 5
Top 100 Contributor
Posts 1,081
Points 11,700
Joined: Jul 2009
Joel H replied on Fri, Jul 12 2013 12:41 PM

Personally, I think it comes down to the fact that AnTuTu isn't a particularly good benchmark, period.

  • | Post Points: 5
Page 1 of 1 (7 items) | RSS