What's actually changed
For most of the web's history, non-human traffic hitting your site was overwhelmingly search engine crawlers, Google and Bing chief among them, indexing your pages so they could show up in search results. That's still happening, but it's now joined by an entirely different category: AI crawlers, operated by the major AI companies, visiting sites specifically to gather content for training models or to answer a live question a user just asked their AI assistant. Independent traffic analyses through 2026 put automated, non-human requests at a substantial and rising share of total traffic hitting the average website, with AI-specific crawlers accounting for a fast-growing slice of that. The exact figures vary depending on who's measuring and how, but the direction is consistent across every source: this isn't a temporary blip, it's a structural shift in what "visiting your website" now includes.
Two different kinds of AI crawler
It's worth separating these into two genuinely different categories, because they behave differently and matter differently to you:
- Training crawlers (bots like GPTBot, ClaudeBot, CCBot, and Bytespider) visit sites in bulk to gather content that gets folded into a model's training data. This tends to happen in large batches, isn't tied to any specific user request, and generally sends no traffic back to you at all.
- Assistant or answer crawlers (bots like ChatGPT-User or PerplexityBot) fetch a specific page in real time because a user asked their AI assistant a question and the assistant needs to check your page to answer it. These can occasionally translate into an actual visitor clicking through to your site when you're cited as a source.
Why the trade feels one-sided
Various industry crawl-log analyses through 2026 have found that AI crawlers, particularly training-focused ones, typically request vastly more pages than they ever send visitors back for, sometimes by ratios in the thousands to one. Compare that to a traditional search engine crawler, which historically sends a real visitor back at a far more favourable rate for every batch of pages it indexes. This is the core tension driving a lot of the current debate: the traffic cost of being crawled is real and ongoing, while the traffic benefit is comparatively rare and, for pure training crawlers, often nonexistent.
robots.txt is a voluntary standard, not an enforcement mechanism. Reputable companies generally respect it, but a crawler that chooses not to simply won't be stopped by the file alone.
What this means for your hosting
Every request, human or bot, consumes real server resources: CPU cycles to generate the response, and processing capacity that's shared across everything else your server is doing at that moment. A generous bandwidth allocation doesn't change the fact that a large volume of automated requests still has to be processed by your server one at a time. On a busy shared hosting environment, an aggressive crawl cycle can be part of what's behind an otherwise unexplained slowdown, alongside the more common culprits we've covered elsewhere, like unoptimised images or too many active plugins.
What you can actually do about it
- Check your traffic logs periodically, available through your hosting control panel, to see whether unusual spikes line up with a specific bot's user agent rather than genuine visitors.
- Decide deliberately, not by default, whether you want your content available to AI training crawlers, AI answer engines, both, or neither. There's a real tradeoff between server load and content protection on one side, and potential visibility in AI-generated answers on the other, and reasonable businesses land in different places on it.
- Use robots.txt to state your preference clearly, understanding it's a request most reputable crawlers honour rather than a hard technical block.
- Keep the rest of your site optimised, since a well-optimised, well-cached site absorbs additional crawl load far more gracefully than one already struggling with unrelated performance problems.
Seeing unusual traffic patterns?
If your site's performance has changed and you can't explain why, our engineers can check what's actually hitting your server.
Frequently asked questions
What is an AI crawler?
An AI crawler is an automated bot, operated by a company like OpenAI, Anthropic, Google, or Meta, that visits websites to collect content, either to train an AI model in bulk or to fetch a specific page in real time to answer a live user question. They're distinct from traditional search engine crawlers, though some companies now run a combined crawler for both purposes.
Does AI crawler traffic cost me anything on shared hosting?
It depends on your plan and how aggressively your site is being crawled. Even where bandwidth is generously allocated, a large volume of automated requests still consumes real CPU and server resources processing each request, which can affect performance for genuine visitors on a busy site.
Does robots.txt actually stop AI crawlers from visiting my site?
It depends on the crawler. robots.txt is a voluntary, honour-system standard: well-behaved crawlers from major companies generally respect it, but it isn't a technical enforcement mechanism, so a crawler that chooses to ignore it can still access your site regardless of what the file says.
Should a small business block AI crawlers?
It depends on your priorities. Blocking reduces server load and keeps your content out of AI training data, but also means your site can't be cited by AI assistants that some customers now use to find businesses. There's no universally correct answer, it's a genuine tradeoff worth deciding deliberately rather than by default.