Microsoft and Yale Get in Nerd Fight Over 'Bing It On' Challenge
The online search wars are serious business, so when Microsoft got called out by a Yale study suggesting the Redmond software giant's claims of users choosing Bing over Google at a nearly 2-to-1 clip in its "Bing It On" challenge are hogwash, Microsoft felt compelled to fire back and address each and every point of the study.
Let's back up a moment. Back in 2012, Microsoft conducted a blind comparison test of Bing versus Google via its Bing It On website, and during that test, almost twice as many people selected Bing's results. Then in 2013, Microsoft updated its claim from saying "People chose Bing web search results over Google nearly 2-to-1 in blind comparison tests" to "People prefer Bing over Google for the web's top searches."
Earlier this week, a Yale law professor published a study in which he and four Yale Law School students conducted their own blind tests on the Bing It On website. His results were far different from Microsoft's, with 53 percent of the participants choosing Google and 41 percent picking Bing; it was a tie among the remaining 6 percent.
"This is not even close to the advertised claim that people prefer Bing 'nearly 2-to-1," wrote Ian Ayres, the Yale law professor.
This prompted a lengthy retort from Matt Wallaert, a behavioral psychologist at Bing, who basically argues that there are several differences in the way Yale and Microsoft conducted their respective studies. One of the things he responded to is the fact that Microsoft doesn't release data from its Bing It On website, and the reason for that is because Microsoft doesn't track it.
"People who come to BingItOn.com are not agreeing to participate in research; they're coming for a fun challenge. It isn't conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it," Wallaert explains. "So we simply don't track their results, because the tracking itself would be incredibly unethical. And we aren't basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific)."
The Yale study also takes issue with Bing It On suggesting search queries, and Wallaert says those suggested searches are from a pool of 500 popular search terms in Google's 2012 Zeitgeist report.
"If Ayres is right and those topics are in fact biasing the results, it may be because we provide better results for current news topics than Google does," Wallaert argues.
Let's back up a moment. Back in 2012, Microsoft conducted a blind comparison test of Bing versus Google via its Bing It On website, and during that test, almost twice as many people selected Bing's results. Then in 2013, Microsoft updated its claim from saying "People chose Bing web search results over Google nearly 2-to-1 in blind comparison tests" to "People prefer Bing over Google for the web's top searches."
Earlier this week, a Yale law professor published a study in which he and four Yale Law School students conducted their own blind tests on the Bing It On website. His results were far different from Microsoft's, with 53 percent of the participants choosing Google and 41 percent picking Bing; it was a tie among the remaining 6 percent.
"This is not even close to the advertised claim that people prefer Bing 'nearly 2-to-1," wrote Ian Ayres, the Yale law professor.
This prompted a lengthy retort from Matt Wallaert, a behavioral psychologist at Bing, who basically argues that there are several differences in the way Yale and Microsoft conducted their respective studies. One of the things he responded to is the fact that Microsoft doesn't release data from its Bing It On website, and the reason for that is because Microsoft doesn't track it.
"People who come to BingItOn.com are not agreeing to participate in research; they're coming for a fun challenge. It isn't conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it," Wallaert explains. "So we simply don't track their results, because the tracking itself would be incredibly unethical. And we aren't basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific)."
The Yale study also takes issue with Bing It On suggesting search queries, and Wallaert says those suggested searches are from a pool of 500 popular search terms in Google's 2012 Zeitgeist report.
"If Ayres is right and those topics are in fact biasing the results, it may be because we provide better results for current news topics than Google does," Wallaert argues.
