Annual Smart Speaker IQ Test Sends Google Assistant To Head Of Class

Loup Ventures has published the results of its Annual Smart Speaker IQ test where it took the four major Smart Speakers on the market that are packed with Alexa, Siri, Google Assistant, and Cortana and put them through a round of tests. The test asked each of the four smart speakers 800 questions to see which of the devices was able to answer the most questions and then ranked the devices based on the percentage of understood queries, and the number of questions answered correctly. The results put Google Assistant at the top of the class.

Google Assistant understood 100% of the aksed queries and was able to answer 87.9% of the questions correctly. Siri came in second place understanding 99.6% of asked queries and answering 74.6% of them correctly. Amazon's Alexa understood 99.0% of asked queries and answered 72.5% of the questions correctly. Rounding out the quartet was Cortana in last place understanding 99.4% of asked queries, which is better than Alexa, but only answering 63.4% of the queries correctly, much worse than Alexa's correct percentage.

The 800 questions asked to the smart assistants included queries designed to test the speaker's ability and utility and were from five different categories. The categories include:

• Local – Where is the nearest coffee shop?
• Commerce – Can you order me more paper towels?
• Navigation – How do I get to uptown on the bus?
• Information – Who do the Twins play tonight?
• Command – Remind me to call Steve at 2 pm today.

Loup notes that it is continuing to modify the question set to reflect the changing ability of smart speakers and assistants. The testing was conducted on second generation Amazon Echo (Alexa), Google Home Mini (Google Assistant), Apple HomePod (Siri), and Harman Kardon Invoke (Cortana) hardware. In the misunderstood queries, Loup points out that nearly every one of them included a proper noun, which was often the name of a local town or restaurant.

While the company tests with commerce-related questions, it believes that the number of people buying things with smart speakers and voice commands is insignificant. That means the questions that were asked are geared more towards product research and local business discovery. Loup also notes that the speakers are improving over time with Google Home gaining 7%, Echo 9%, and Cortana 7% over the last year. Siri increased 22% over nine months, and the gain was the result of the enabling of more domains in the past year.