AI Models Trusts Its Sources Too Much

One thing I do not like about Google's AI Overview is the fact that it relies too much on the internet's information.

If you were to ask it about something extremely niche and lacking in information, chances are, it's going to regurgitate what is already there and hallucinate even further. Worse, if the available information that it can take in is bad, e.g., misinformation, it'll state and elaborate as if they were factual.

I've seen this play out more times than I'd like it to, I mean, you could go ahead and do it right now! Ask it about some random new JavaScript framework and its capabilities, or the brand new project you've just uploaded to GitHub; the Overview confidently weaves together the thin scraps it finds online and makes it sound as authoritative as possible.

The First Answer

Another issue is that I believe the result that it gives you at first is run with a less capable LLM model due to the sheer number of users on Google; it is only until you go to their AI Mode tab, or ask it further in the prompt box, that it leverages a better model to help you.

This makes sense from a cost perspective, sure. Google is serving AI Overviews to what, billions of queries a day? Running a frontier-level model on every single one of them would be financially and computationally brutal, even for a company with Google's infrastructure. I don't even think Elon would dare shell out his wealth for this, subsidization can only go so far. Hence, the default experience is optimized for speed and cost, using a lighter model that is fast and cheap but shallower in its reasoning.

Basically, the amount of reasoning that it can do to give a quality and factual answer from the very start is immensely dependent on the quality of the available sources. A smart model may be able to interpolate and reason itself to understand whether the sources are reliable, and if it is enough to confidently give out an answer from them, but even then, that has its limits. For these 'instant' models though? No way, Jose.

Is This Even Fixable?

Now, is it a fully fixable issue? Probably not. The internet is a giant network, and there can only be so many people verifying and publishing content for Google to index. There is a hard ceiling here, you cannot summarize your way out of a topic that genuinely has two articles and a forum thread written about it. No amount of model improvement or internal knowledge cramming changes the fact that some corners of human knowledge are just thinly documented, and always will be.

Better AI models down the line will be able to reason better from the limited amount of sources it has for a cheaper cost, especially for logic problems such as math, but even that is a tough line to cross considering the user base of Google.

Improvements are coming, that much is inevitable (have you seen DeepSeek V4 Flash), but "better reasoning on scarce data" and "cheap enough to run on every single search" are two goals that pull in opposite directions. Google's scale is both its greatest asset and its biggest constraint here. A smaller AI product can afford to run its best model on every query, but Google? They hold like what, 90% the world's search engine market share? They do not have the luxury of tanking the cost without the shareholders freaking the hell out.

Oh, and Autonomous AIs too

It doesn't help either that there are autonomous AI agents out there making posts on their own, which, of course, isn't something known for being reputable and verifiable, further muddying the waters for Google's AI Overview. This might end up being the bigger long-term threat, honestly.

Misinformation from humans is bad enough, but it is at the very least bounded by how much time and effort people are willing to put in. Autonomous agents do not have that constraint, however. They can generate content at a scale and speed no human misinformation campaign ever could! I myself am already tired of the barrages of "pull request slops" from them, and I do not want it anywhere else.

If Google's indexing pipeline starts absorbing a meaningful percentage of AI-generated, unverified content as "source material," it'll be like Ouroboros, and that can get ugly fast...

Where This Leaves Us

That being said, I'm fairly interested in how this issue will be resolved in the future (if ever, haha 🫠), as AI technology moves really fast and something considered impossible today may just be possible next week. Maybe the answer isn't a smarter model at all, but perhaps better provenance tracking, some way of weighting sources by verified trustworthiness rather than just relevance or recency. Maybe just a refusal to display an overview if the sources simply aren't enough for the model to be confident. Or maybe it's something nobody's thought of yet, which, given how fast this space moves, wouldn't surprise me one bit.

For now, though, the practical takeaway is pretty simple. The more niche your question, the more you should probably disregard the AI Overview.