Nvidia's Kal-El Demonstration Marred By Benchmark Confusion

rated by 0 users
This post has 8 Replies | 1 Follower

Top 10 Contributor
Posts 25,861
Points 1,172,065
Joined: Sep 2007
ForumsAdministrator
News Posted: Mon, Feb 21 2011 4:01 PM
When Nvidia announced its next-generation Tegra product at the Mobile World Conference, it pulled out all the stops in an effort to impress. The stats themselves were impressive—the chip packs a twelve-core GeForce GPU in with a quad-core ARM CPU—but NV opted to hammer home the point by showing benchmark results.

According to Nvidia, Kal-El turned in a score of 11,352 in the embedded processor benchmark Coremark while a T7200 Core 2 Duo (65nm, 2GHz dual-core, 4MB L2, 667MHz FSB) returned a score of just 10,136. If you want to see Nvidia's original video, you can do so here. We weren't completely sold on the results because, as we explained:
The program [Coremark], published by the Embedded Microprocessor Benchmark Consortium (EEMBC), is only designed to test the core functions of a processor. According to CoreMark.org: "It is encouraging to see the industry, as well as academia, adapting to a new standard so quickly, but let us not forget – CoreMark only targets core operations. EEMBC’s full-featured application benchmarks are much better suited for testing a processor’s capability in a real application. Furthermore, processors are becoming increasingly complex and one core-based benchmark is insufficient for a comprehensive analysis.
It turns out we were right. The credit for this discovery goes to ilsistemista.net, who caught the very different fine print under Kal-El vs. the T7200.



It turns out that the version of Coremark that Kal-El ran was heavily optimized and used a relatively new version of GCC, while the T7200 flavor was compiled using an old version of GCC with minimal optimization. The next step Ilsistemista took was to test what happens when Coremark is compiled and run on the T7200 using the same optimizations that were used for Kal-El. As you'll see, there's a bit of a difference.


When the scales are evenly balanced the T7200 turns in results nearly 50 percent faster than NV's published numbers.

What Was The Point?

The strangest thing about Nvidia's move is that there's no reason for it. Compare Kal-El to Tegra 2, and you'll note that Kal-El is no less than 94 percent faster than its predecessor. It wouldn't surprise us if Kal-El's graphics engine really is 5x faster than T2's, at least in some metrics. Certainly no one else is seriously talking about a mobile platform that can push 2560x1600 dropping by the end of the year. There's no reason to think Kal-El will be inherently limited to larger devices, either—NV should be able to fit the part into smaller power envelopes by disabling CPU/GPU cores. If a quad-core / 12-pipe configuration works well for a tablet, a dual-core / 6-pipe configuration should work just fine for a smartphone.

Second, there's no single test that's emerged on a consumer level as "the" benchmark for smartphones. Various browser-centric timed tests come closest, but browsers themselves are a significant confounding variable. An awful lot of testing boils down to whether or not Phone A "feels" faster than Phone B. Unfortunately, research has shown that human beings are pretty bad at actually judging such improvements and will often think that a newer / supposedly faster device actually is faster when timed tests prove it isn't.

Third, and perhaps most importantly, this sort of strategy inevitably raises questions about why NV felt it had to skew the numbers to make Kal-El look good. Kal-El, to our way of thinking, still looks pretty darn good on its own. If it delivers as promised, it'll be a huge leap from Tegra 2. As for its theoretical performance against a five year old Core 2 Duo that would never fit inside a tablet, we don't see a reason to care. 
  • | Post Points: 80
Top 100 Contributor
Posts 988
Points 10,635
Joined: May 2010
Location: Toronto Baby!
coolice replied on Tue, Feb 22 2011 9:35 PM

This article essentially confirmed what what most of the members commented in last weeks article when nvidia first demoed this "next gen "kal-el" mobile processor"... goes to show that companies like to play us a fool, well at least the general public!

  • | Post Points: 20
Top 10 Contributor
Posts 4,821
Points 45,685
Joined: Feb 2008
Location: Kennesaw
rapid1 replied on Wed, Feb 23 2011 1:57 AM
OS:Win 7 Ultimate 64-bit
MB:ASUS Z87C
CPU:Intel(R) Core(TM) i7 4770 ***
GPU:Geforce GTX 770 4GB
Mem:***ingston 16384MB RAM
  • | Post Points: 20
Top 500 Contributor
Posts 182
Points 2,085
Joined: Dec 2010
Location: California

Hit the nail on the head. So unnecessary to compare your latest and greatest by inflating numbers when you can easily run on the platform of "its way way better than our last product". C'mon  now green!

"The old appeals to racial, sexual, religious chauvinism to rabid nationalist furver are beginning not to work. A new consciousness is developing which sees the earth as a single organism and recognizes that an organism at war with itself is doomed"- Carl Sagan

  • | Post Points: 5
Top 500 Contributor
Posts 159
Points 1,365
Joined: Oct 2009
Schmich replied on Wed, Feb 23 2011 6:19 AM

This is so pathetic on Nvidia's part. I don't mind too much descriptions that their products are the best but lying about performance? So sad.

  • | Post Points: 20
Top 10 Contributor
Posts 8,577
Points 103,200
Joined: Apr 2009
Location: Shenandoah Valley, Virginia
MembershipAdministrator
Moderator
realneil replied on Wed, Feb 23 2011 9:46 AM

coolice:
companies like to play us a fool

And they never seem to learn that they usually get caught with their pants down when they try this kind of BS.

Case in point: Sony and their infamous root-kit embedded onto millions of their Music CD's.

The fact is that there's a whole lot of very smart, connected people out there that are looking hard at what these companies are doing. Getting away with 'slight-of-hand' shenanigans is not an option these days.

Here's a 1956 Volvo that looks almost new.


Dogs are great judges of character, and if your dog doesn't like somebody being around, you shouldn't trust them.

  • | Post Points: 5
Top 100 Contributor
Posts 1,087
Points 11,005
Joined: Jun 2010
Location: Pennsylvania
CDeeter replied on Wed, Feb 23 2011 12:16 PM

Wow they just lost a lot of respect in the tech community. Did they do this just to take a pot shot at Intel? Or were they trying to say that they can bring desktop class power to SOC devices? Either way they blew it.

  • | Post Points: 5
Top 200 Contributor
Posts 324
Points 4,305
Joined: Nov 2010
Location: Perth Amboy, New Jersey

i7-950@3.8 + Asus Sabertooth X58+ CM692 Advanced+ 6GB A-Data Gaming DDR3@1600

ATI Radeon 6870+ 1TB Hitachi HD+ 750w Modular PSU + Asetek Liquid Cooler + DigitalStorm PC  

YesThanks To HotHardware.ComYes

  • | Post Points: 5
Top 50 Contributor
Posts 3,236
Points 37,910
Joined: Mar 2010
AKwyn replied on Thu, Feb 24 2011 5:04 AM

Eh. This happens mostly every day. I don't blame them, they wanted to make their processor more powerful then it was.

While it is unnecessary and laughable that they would do this, they haven't lost me as a customer just yet. They have to do something much worse in order to lose me.

 

"The future starts with you; now start posting more!"

  • | Post Points: 5
Page 1 of 1 (9 items) | RSS