Apple Calls Out Consumer Reports For Disabling Hidden Safari Cache Setting In MacBook Pro Battery Testing
“For instance, in a series of three consecutive tests, the 13-inch model with the Touch Bar ran for 16 hours in the first trial, 12.75 hours in the second, and just 3.75 hours in the third,” wrote Consumer Reports at the time. “The 13-inch model without the Touch Bar worked for 19.5 hours in one trial but only 4.5 hours in the next. And the numbers for the 15-inch laptop ranged from 18.5 down to 8 hours.”
Now, it’s coming to light exactly how the reviewers got such widely differing results across all three models that it tested. According to Apple, Consumer Reports disabled a hidden setting in Safari, which in effect turned off the browser cache. According to Apple, disabling this setting (which regular consumers wouldn’t even use) gave battery performance results that were not indicative of what users would see in the real world.
In a statement, Apple explained:
Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test. This is the best pro notebook we’ve ever made, we respect Consumer Reports and we’re glad they decided to revisit their findings on the MacBook Pro.
By using the setting meant for developers, it looks like a bug was triggered that explained the poor battery performance. For its part, Consumer Report has responded with a blog posting on the matter, writing, “We turn off caching as part of Consumer Reports' standard laptop test protocol. Caching is a feature used by many computers to store Web pages locally on a hard drive for faster retrieval by the browser.”
Consumer Reports goes on to state that it disables browser cache for every system that it tests (including Windows machines), as it is the only way that it can reproduce reliable battery life figures. It even points out that its tests “are not designed to be a direct simulation of a consumer’s experience.”
The publication writes:
Many of these settings are set by default to extend battery life. That’s generally a good thing. But because these settings are so variable and situation-dependent, we turn several of them off during testing. For instance, we turn the screen auto-dimming features off on all laptops and set the displays to a constant level of brightness…
We also turn off the local caching of web pages. In our tests, we want the computer to load each web page as if it were new content from the internet, rather than resurrecting the data from its local drive. This allows us to collect consistent results across the testing of many laptops, and it also puts batteries through a tougher workout.
We can definitely understand Consumer Reports’ reasoning for wanting to level the playing field for all of the notebooks that it tests, and it appears that the bug that was discovered as a result of disabling the cache was at least one of the culprits for the battery discrepancies. The publication went on to acknowledge, “Indeed, when we turned the caching function back on as part of the research we did after publishing our initial findings, the three MacBooks we’d originally tested had consistently high battery life results.”
So, what do our HotHardware readers think about these latest revelations? Who’s in the right?
