The world of speech technology and personal voice assistants is picking up quickly and has an incredible amount of potential for future innovation. Siri isn’t the only player in this market anymore and Apple, Microsoft and Google all have their sights on revolutionizing what is possible with speech technology.
There are many reviews out there comparing Siri, Google Now, and Cortana, but as a language technologies company, we decided to do things a little differently. We put Siri, Cortana, and Google Now to the test in five languages to see which voice assistant has the best multilingual chops. Our savvy multilingual team asked 18 questions in English, Italian, French, German and Mandarin.
We gave each question a possible score of 1.0, 0.5 and 0.0. Answers that scored a 1.0 gave our testers a quick and easy correct response. 0.5 answers weren’t quite as straightforward and came back with a list of search results, where somewhere within the clutter, we found the correct answer. 0.0 responses did not understand our questions, gave irrelevant search results, or by their own admission, stated that they could not complete the task.
First up was English. Not surprisingly, English had the highest accuracy rates across all assistants. Siri scored on top (14.5/18) but both Google Now (11.5) and Cortana (12.5) weren’t trailing too far behind. We thought this was going to be a pretty even match and were especially impressed with the newest player on the block- Cortana. So with round one complete, we noted two key findings… none of these assistants can mute your phone or guide you to success on the stock market (spoiler- in English or any other language).
Next came Italian. Congratulations to Siri, who unequivocally left her competitors in the dust. Apple’s localization team should be proud because Siri scored identically to English (14.5/18). Google Now, on the other hand, scored a less than impressive 8.5/18 and our emerging star was suddenly not so remarkable anymore, scoring 6/18. A quick caveat for all of you Windows Phone users- we were running the developer release of Cortana in beta for Italian, French, German and Mandarin. We’re convinced that that with some proper localization (or maybe some practice with Duolingo), Cortana could rise to the top.
French and German results were similar to Italian. Siri dropped slightly in German and French and Google Now and Cortana improved marginally in both languages. Up until now, the only two commands that could be completed in every language were making phone calls or sending text messages. Apple, Microsoft, Google… we still have some work to do.
Last came Mandarin. We suspected that this might be the deciding factor that would set apart the boys from the men. Siri, again, scored on top with a 13.5 (only 1 point below its English score). Cortana scored a low 5, which given it’s performance in Italian, German & French, isn’t so surprising. Google, however, averaging a score of about 10/18 until now, dropped to a disconcerting score of 1.5/18. Google offers over 50 languages and dialects, far exceeding both Siri or Cortana. It is clear that many of these languages (like Mandarin) are only setup for voice-enabled search and that they do not yet have the capability to act as a full voice assistant. Having said that, only 3 out of our 18 questions scored as high as a 0.5, meaning that the ability to use Google simply for voice-enabled search in Mandarin is questionable at best. A 56% decrease in accuracy between English and Mandarin should still be cause for concern for the Google team.
In conclusion, Siri won this battle, Cortana was voted “most likely to go to web search” and Google Now needs to go back to college and pick up a few Mandarin classes.
0 comments:
Post a Comment