Google’s Hindi AI Overviews: A Need for Quality Improvement

Given India’s language diversity, digital content companies already face a challenge in trying to show and translate content accurately. Google is facing a similar problem with AI Overviews recently rolled out in the country.

The company introduced Hindi support for AI Overviews in the country earlier in August. Users in India can also switch between Hindi and English without leaving the search page. But the feature is disappointingly inconsistent.

One simple example was a result we picked from Google’s blog. The GIF showed an answer for “Cheeni ki jagah chai mai kya daal sakte hai?” which translates to “What is a substitute for sugar in tea?” During testing, we changed up the order of the words to “Chai mai cheeni ke jagah kya daal sakte hai?” and Google simply didn’t show an answer.

Another problem we spotted was translating English words literally. When we asked in Hindi, “What kind of food can we eat during summer?” one of the answers was “Chiknai wali cheezien,” which translates to “Sticky things” — and that’s confusing. When I switched over to English, Google showed me “Oily” an an option, which is a very different thing from “sticky” when it comes to food. Either way, it’s kind of a weird suggestion.

In another instance, when we asked about YouTube’s ownership, AI Overviews said “Until 16 February 2023, Neal Mohan was Google’s CEO” in Hindi, which is incorrect. The English text had the correct version, saying, “As of 16 February 2023, Neal Mohan is Google CEO.”

There are also qualitative problems. When we asked, “When is Diwali this year?” instead of getting a simple answer, we got a paragraph on last year’s Diwali, then a carousel of links, and then the actual answer we were looking for.

When we asked if one can eat food with spices in both Hindi and English multiple times, we got inconsistent answers through AI Overview.

However, there are more concerning results related to other topics like menstruation and pregnancy. For one of our questions, about when someone should think about having kids after marriage, the overview answer’s first paragraph said couples should wait for at least two years, in confusing Hindi.

It also mentioned that if someone gets married around the age of 25, they have “three years” — without specifying for what. This paragraph was apparently sourced from a slideshow article in Hindi on an Indian news site, which had written the info based on various opinions on Quora and other blogs.

When we asked about what food to eat when someone is menstruating, some of the answers suggested “Drinking milk with many things” and lemon for mood swings.

Google’s AI Overview also misses context. When we asked about food places in Delhi, it suggested that Bangla Sahib Gurudwara, a place of worship for Sikhs, is open round the clock and you can get tea and Indian snacks like Samosa and Kachori. However, this fails to mention something important that the source article did: that you could get these items outside the Gurudwara, an operative word that the AI tool missed while summarizing. The snacks are not in the place of worship.

Some of the results for similar questions in English are much better than the Hindi ones. This could be partially due to more and better sources available in that language. But some of the problems we highlighted were due to Google systems’ mistakes, such as out-of-context summaries and inaccurate translations. Google has worked on search and language problems in India, and the expectation is that it should have delivered a better outcome at its first go.

Google said that its AI Overviews only appear for queries where it has a high confidence in the quality of the output, saying the feature is “rooted in our core search quality systems” and will “only show information that’s backed up by top web results.

“Our tests show that the accuracy rate for AI Overviews is on par with other features like Featured Snippets. When issues arise, they may be the result of our systems misinterpreting web content or reflecting inaccuracies on the web — and we use these examples to improve, as we do with all Search features,” a Google spokesperson said. 

Google can blame the quality of the sources that it is citing. However, it’s Google’s algorithm that decided that the question and sources are worthwhile to be summarized by AI. Not all users will look at the material on source sites and merely point at Google for displaying inaccurate or incomprehensible answers. 

AI Overviews has been a disappointing endeavor for Google. Earlier this year, the company caught a lot of flack for surfacing answers from Reddit telling a user to add glue to pizza. AI Overviews told another user to eat “one small rock per day,” an answer that was sourced from The Onion. Many of the examples we tested in Hindi simply didn’t provide correct information about the topic.

India has more than 830 million internet users, and a good chunk of them use Google for search. If the company wants more folks to use AI Overviews in local languages, it will need to step up its game in terms of language and content accuracy.