AI Is Taking Your Content but Sending Back Almost Nothing

By Justin Aldridge · Updated

In short

AI tools such as ChatGPT, Gemini and Claude crawl websites to train their models and answer users, but send very little traffic back: a Known Agents study found only 0.1% of human traffic comes from AI chat referrals. Blocking them means losing the chance to be cited, so the practical choice is to allow the AI crawlers that can help you and block the ones that only add load.

Illustration of web pages and documents funnelled into an AI machine, with only a trickle coming out the other side

It’s no secret that AI tools such as ChatGPT, Gemini and Claude crawl websites for information to train their models on, and then use that information to generate answers for their users, but ultimately don’t pass on that traffic back to the websites where the information initially came from.

What the study found

A recent study by Known Agents reveals that one-third of all web traffic is now bots, with 28% of those being from AI scrapers and agents. You may think that your website traffic is looking healthy, but a big portion of that traffic may be bots.

The study also revealed that only 0.1% of human traffic actually comes from AI chat referrals, meaning that websites are being heavily crawled to feed AI answers, but the click-back traffic is minuscule.

As the competition for AI increases, so will the number of AI crawlers hitting your website, and their frequency and depth.

To block or not to block

This raises the question as to whether it’s worth allowing AI bots to aggressively crawl (and steal) your website content. The dilemma comes from wanting to be mentioned and cited by AI tools, even if the traffic is negligible, and the morality of the whole thing in that these tools are benefiting from your hard work in creating your unique content, and then not getting rewarded for it.

The reality is that we are in a “damned if you do and damned if you don’t” situation.

Personally, I am allowing all of my websites to be crawled and indexed by AI bots, although limiting this to the ones that are actually potentially helpful, such as ChatGPT, Gemini, Claude, Perplexity, etc. There are many other bots, such as Bytespider, AHREFS, etc., which probably don’t have any value and are worth blocking to reduce the load on your website.

Autonomous AI agents growing

One other area of potential interest from the study was the rise in AI agents that actually navigate like a human (Manus, Google’s agent, ChatGPT Agent). These agents are “sent off” to take actions on websites, and although this traffic is still relatively low, the study shows that average pages-per-session jumped 93% and session duration 38% in 90 days. That’s a significant increase in just a few months.

This is definitely where we are heading, agents will do much of the leg work for the user and a lot of research time will be spent in AI tools themselves. The website will become just where the user takes the final action or is the final true source for the AI and the user.

Whether we like it or not, AI tools are stealing our content, sending little back and AI agents will spend more time on our websites doing things which don’t exactly match how a real user would navigate the website. It’s all changing and changing fast!

We use analytics cookies (Google Analytics) to understand how the site is used. They only run if you accept. Cookie policy