
A few years ago, asking an AI model a question was slow and expensive enough that most businesses treated it as a novelty. That changed quickly. Stanford's 2025 AI Index estimated that the cost of running a model at roughly GPT-3.5-level performance fell from about US$20 per million tokens in November 2022 to about US$0.07 by October 2024, a drop of more than 280-fold in under two years. When something becomes that much cheaper and faster, it ends up everywhere, including on your website. Most commentary on faster AI focuses on the people using chat tools. Far less attention goes to the other end of the connection. Every AI answer that cites a page, every model refreshed with new training data, and every agent that checks a price or reads a returns policy on someone's behalf starts with a machine requesting a URL from a server. More and more often, that server is yours. The scale is no longer marginal. The 2026 edition of the Thales Imperva Bad Bot Report, covering full-year 2025, found that automated traffic made up about 53% of web traffic, up from 51% the year before, leaving human visitors at roughly 47%. Not all of that automation is AI, and not all of it is harmful. The direction, though, is unmistakable: a growing share of the requests your site answers come from software rather than people. That changes the question site owners should be asking. It used to be enough to ask whether a page loads quickly for a visitor in Halifax or Kelowna. Now the question is whether your hosting can absorb a steady background load of crawlers, fetchers and agents and still respond quickly to humans, to search engines, and to the AI systems deciding whether your content gets cited at all. This guide explains what has changed, what AI crawler traffic actually costs a website, how it connects to search visibility and indexing, and what to look for in a hosting environment that now has to serve both audiences. What is AI crawler traffic? AI crawler traffic is the portion of a website's requests that comes from automated systems collecting or retrieving content...



















