Majority of Search Engines Face Reddit’s Restrictive Access Policy

Reddit is one of the internet’s top places for discussion, and it’s an important bastion of internet culture. Which is also the reason why it often shows up near the top of search results. Now, though, it looks like Google is the only search engine that can properly pull up Reddit search results.

Reddit has seemingly restricted access to its content , effectively making Google the only search engine capable of displaying Reddit results. Alternative search engines like Bing, DuckDuckGo, and others are now unable to properly crawl Reddit and display new content. Some of these search engines might show some non-descript Reddit search results, while others show none at all. While neither Reddit nor Google have officially commented on this matter, this may be related to Reddit’s multi-million dollar deal with Google to scrape Reddit data for AI training purposes.

The decision has drawn criticism from smaller search engine providers. Colin Hayhurst, the CEO of search engine company Mojeek , told 404 Media, “They’re [Reddit] killing everything for search but Google.” Microsoft (which owns Bing), DuckDuckGo, and other search engine companies have not chimed in yet.

Reddit’s actions come amidst a broader trend of websites blocking bots used by AI companies for data scraping. The platform recently updated its robots.txt file , a set of instructions for web crawlers, to strictly prohibit any automated access. The current robots.txt file is set to disallow all web crawling, including regular search engines (an archived version from July 23rd does not). Google, owning the most popular search engine out there, is in an unique position to train its generative AI on its search results and on the other data it discovers. We can’t think of a single generative AI developer that has trained its models in an “ethical” manner, but Google might be willing to go way lower than others.

Source: 404 Media