Futuremark Bans HTC One M8 From 3DMark For Android Benchmark Results, HTC Defends Device's Behavior
Futuremark isn't letting up on its decision to publicly shame handsets and device manufacturers that get caught trying to game the company's 3DMark benchmark. Just the opposite, Futuremark wasted no time calling out HTC and its recently refreshed One (M8) for the way the device handles itself when running 3DMark .
One of the rules manufacturers must follow is they can't implement a detection scheme that lets the device know when 3DMark is being launched, nor can the device alter its normal behavior based on such detection. This is where the HTC One (M8) gets itself in hot water.
The HTC One (M8) kicks up its CPU frequency about 15 percent on average when running 3DMark. Futuremark observed this by comparing the behavior of the phone when running a publicly available version of 3DMark downloaded from Google Play versus a renamed but otherwise identical version of the benchmark.
As Futuremark has done with other devices that run afoul of the rules, it delisted the HTC One (M8) from its 3DMark for Android rankings and issued a press releases calling the company out. However, what's interesting is that HTC sees this as a feature rather than cheating.
"Benchmarking tests look to determine maximum performance of the CPU and GPU and, similar to the engine in a high-performance sports car, our engineers optimize in certain scenarios to produce the best possible performance. If someone would like to get around this benchmarking optimization there are ways to do so, but we think most often this will not be the case," HTC explained to CNET in an email.
Futuremark is aware of HTC's stance, but doesn't like the explanation. According to Futuremark, HTC's viewpoint that benchmarks look to determine max performance of core parts is a "common misconceptions." The purpose of 3DMark, and all of Futuremark's benchmarks, is to measure real-world performance, the company counters.
"To do that, a device must run the benchmark as if it were any other application," Futuremark explains. "When a device detects 3DMark, and modifies its behavior as a result, the benchmark scores are no longer accurate and cannot be used to make fair comparisons between devices."
HTC says it's planning to implement a High Performance Mode on U.S. handsets. When enabled, the phone would run faster in certain situations, such as 3DMark and games in general. If that's the case, it further muddies the water between what constitutes real-world performance versus cheating.
One of the rules manufacturers must follow is they can't implement a detection scheme that lets the device know when 3DMark is being launched, nor can the device alter its normal behavior based on such detection. This is where the HTC One (M8) gets itself in hot water.
The HTC One (M8) kicks up its CPU frequency about 15 percent on average when running 3DMark. Futuremark observed this by comparing the behavior of the phone when running a publicly available version of 3DMark downloaded from Google Play versus a renamed but otherwise identical version of the benchmark.
As Futuremark has done with other devices that run afoul of the rules, it delisted the HTC One (M8) from its 3DMark for Android rankings and issued a press releases calling the company out. However, what's interesting is that HTC sees this as a feature rather than cheating.
"Benchmarking tests look to determine maximum performance of the CPU and GPU and, similar to the engine in a high-performance sports car, our engineers optimize in certain scenarios to produce the best possible performance. If someone would like to get around this benchmarking optimization there are ways to do so, but we think most often this will not be the case," HTC explained to CNET in an email.
Futuremark is aware of HTC's stance, but doesn't like the explanation. According to Futuremark, HTC's viewpoint that benchmarks look to determine max performance of core parts is a "common misconceptions." The purpose of 3DMark, and all of Futuremark's benchmarks, is to measure real-world performance, the company counters.
"To do that, a device must run the benchmark as if it were any other application," Futuremark explains. "When a device detects 3DMark, and modifies its behavior as a result, the benchmark scores are no longer accurate and cannot be used to make fair comparisons between devices."
HTC says it's planning to implement a High Performance Mode on U.S. handsets. When enabled, the phone would run faster in certain situations, such as 3DMark and games in general. If that's the case, it further muddies the water between what constitutes real-world performance versus cheating.
