AI Is Becoming Speedier

A few years ago, asking an AI model a question was slow and expensive enough that most businesses treated it as a novelty. That changed quickly. Stanford's 2025 AI Index estimated that the cost of running a model at roughly GPT-3.5-level performance fell from about US$20 per million tokens in November 2022 to about US$0.07 by October 2024, a drop of more than 280-fold in under two years. When something becomes that much cheaper and faster, it ends up everywhere, including on your website.

Most commentary on faster AI focuses on the people using chat tools. Far less attention goes to the other end of the connection. Every AI answer that cites a page, every model refreshed with new training data, and every agent that checks a price or reads a returns policy on someone's behalf starts with a machine requesting a URL from a server. More and more often, that server is yours.

The scale is no longer marginal. The 2026 edition of the Thales Imperva Bad Bot Report, covering full-year 2025, found that automated traffic made up about 53% of web traffic, up from 51% the year before, leaving human visitors at roughly 47%. Not all of that automation is AI, and not all of it is harmful. The direction, though, is unmistakable: a growing share of the requests your site answers come from software rather than people.

That changes the question site owners should be asking. It used to be enough to ask whether a page loads quickly for a visitor in Halifax or Kelowna. Now the question is whether your hosting can absorb a steady background load of crawlers, fetchers and agents and still respond quickly to humans, to search engines, and to the AI systems deciding whether your content gets cited at all.

This guide explains what has changed, what AI crawler traffic actually costs a website, how it connects to search visibility and indexing, and what to look for in a hosting environment that now has to serve both audiences.

What is AI crawler traffic?

AI crawler traffic is the portion of a website's requests that comes from automated systems collecting or retrieving content for artificial intelligence products. It includes crawlers that gather pages to train models, crawlers that build the indexes AI search features cite, and fetchers that retrieve a page in real time because a user asked an assistant to read it.

Unlike most human visits, AI crawler traffic often reaches deep, rarely visited pages. That makes it more expensive to serve than its raw volume suggests, and it is why the same number of requests can feel very different on the server depending on who is sending them.

Why faster AI changes the traffic your server sees

Cheaper inference means more machine visits

When an AI query cost real money, developers rationed it. When it costs a fraction of a cent, they build products that make dozens of calls per task: searching, fetching pages, comparing sources, re-checking facts. Speed compounds the effect. A model that responds in a second can afford to go and read the web live in the middle of an answer instead of relying only on what it learned during training.

The result is not one new crawler. It is an entire layer of automated visitors with different purposes, schedules and manners.

The three kinds of AI traffic you will find in your logs

Separating AI-related requests by purpose matters, because the right response to each one is different.

Training crawlers collect pages in bulk to build or refresh the datasets used to train AI models. They tend to crawl broadly and repeatedly, and a visit from one rarely sends a human back to your site. OpenAI's GPTBot is an example; the company documents that disallowing GPTBot signals that a site's content should not be used to train its foundation models.

AI search and indexing crawlers build the index that an AI answer feature draws on when it cites sources. These behave much like a traditional search engine crawler: they visit to keep an index fresh, and the payoff for the site is visibility and referral traffic. OpenAI's OAI-SearchBot fills this role for ChatGPT's search features, and the company states that sites opted out of it will not be shown in those search answers. Google is different: it says AI features such as AI Overviews and AI Mode are part of Search and are governed by ordinary Googlebot access, so there is no separate AI crawler to manage for them.

User-triggered fetchers retrieve a specific page because a person asked an assistant to do something, such as summarise an article, check a product page or compare two services. These requests arrive one at a time, often in real time, and look more like a visit than a crawl. OpenAI notes that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply to them.

A fourth group sits alongside all three: bots that impersonate legitimate crawlers. Any client can claim any user-agent string, so a request labelled with a well-known crawler name is not proof of who sent it. The major operators publish IP ranges or verification methods for exactly this reason, and that matters when you decide what to allow.

Taking far more than they send back

Traditional search crawling worked as a rough trade: a search engine fetched your pages, and in return it sent you visitors. AI crawling has stretched that trade considerably. Network-level measurements published by one of the largest CDN operators found that in June 2025, Google crawled websites about 14 times for every referral it sent, while leading AI operators were crawling somewhere between roughly 1,700 and 73,000 times per referral. The same operator reported that training accounted for close to 80% of AI crawling by mid-2025.

Those ratios shift from month to month, and some AI operators have narrowed the gap as their search products have grown. The underlying point holds regardless: a great deal of AI crawling is not fetching your page to show it to someone right now. Your server still does the work.

What AI crawler traffic actually costs a website

Bandwidth, CPU and the uncached request

A crawler that requests a static image or a cached HTML file costs very little. A crawler that requests a dynamically generated page costs much more, and that is where most of the strain comes from.

On a typical WordPress site, a request that misses the page cache wakes up PHP, runs database queries, assembles the page and sends it. Human visitors mostly land on a handful of popular pages that are already warm in the cache. A broad crawler does the opposite. It walks the long tail of the site: old posts, tag archives, paginated listings, internal search results and filtered product views. Those are exactly the pages least likely to be cached, and every one of those requests is real compute.

Faceted navigation and internal search are the worst offenders. A store with ten filters can expose thousands of URL combinations that each render a slightly different page. No person browses them all. A crawler might try.

Slower responses for the people who matter

Shared hosting environments allocate CPU, memory and concurrent processes per account. When a burst of uncached bot requests saturates those limits, the next human request waits in the queue behind them. The visitor never learns that a crawler is the reason. They simply see a slow page, or a timeout at checkout.

That is the practical risk of unmanaged AI crawler traffic for a small or mid-sized business. It rarely looks like a dramatic outage. It looks like a gradual rise in response times during busy periods that quietly erodes conversions and the experience visitors have on the site.

The knock-on effect on Google's crawling

There is a second-order effect that matters a great deal for indexing. Google's documentation describes a crawl capacity limit, which it also calls hostload: a cap on how much connection time Google will use on a site so it does not overload the server. Google states that this limit rises when a site responds consistently and its response times, including latency and time to first byte, stay stable or improve. It falls when the site slows down or returns server errors or rate-limiting signals such as HTTP 429.

Google's documentation also says the crawl capacity limit is shared across all of Google's crawlers, so heavy demand from one can reduce what is available to the others. That statement is about Google's own crawlers, not third-party bots. The mechanism behind it is what matters here. If unmanaged bot traffic pushes your response times up, Google measures a slower, less healthy server, and the logical consequence is that it may crawl you less. For a site trying to get new or rewritten pages discovered and evaluated, that is the wrong direction. Our Core Web Vitals guide [PLACEHOLDER: Core Web Vitals guide URL] covers the front-end side of the same speed story.

Put simply, slow hosting no longer costs you only impatient visitors. It can also cost you crawl attention from the search engine whose index still sends most referral traffic.

Speed is now an AI-visibility issue, not just a user-experience metric

Google's AI features run on ordinary crawling

Google's guidance for site owners on AI Overviews and AI Mode is plain: there are no additional technical requirements to appear in those features beyond normal SEO fundamentals. Among those fundamentals, Google lists making sure crawling is allowed in robots.txt and by any CDN or hosting infrastructure.

That last clause deserves a second reading. Google is explicitly acknowledging that hosting-level settings, such as firewall rules, bot-blocking features and rate limits, can stop its crawler even when robots.txt allows it. A blunt "block all bots" switch on a server or CDN can remove a site from the very features its owner is trying to appear in.

Blocking the wrong bot hides you; blocking nothing exposes you

Many site owners respond to AI crawling with an all-or-nothing reflex. Both extremes cause problems.

Blocking everything that looks like AI can remove you from AI-assisted search results. With OpenAI, for instance, opting out of OAI-SearchBot keeps a site out of ChatGPT search answers, while opting out of GPTBot only signals that content should not be used for model training. With Google, the Google-Extended token controls whether content can be used for training and grounding in Google's Gemini products, but Google states that it does not affect inclusion in Search, including AI features there. Google-Extended is a robots.txt token rather than a separate crawler, so disallowing it does not reduce the number of requests your server receives.

Allowing everything without limits has the opposite problem. Your server carries every training crawl, scraper and impersonator at full rate, and the cost lands on your human visitors and on Googlebot's view of your server's health.

The workable middle ground is a policy based on purpose, which the sections below lay out.

Pages that machines can read cleanly

Speed is not only a server question. A page whose main content appears only after client-side scripts run is harder and more expensive for any automated system to process, and not every automated fetcher renders JavaScript the way a modern browser does. Serving the core text of important pages as HTML from the server, with a page cache doing most of the work, makes those pages cheaper to fetch and more reliably readable by search engines, AI systems and people on slow mobile connections alike.

How to tell whether AI bots are already straining your site

Before changing anything, measure. Most sites that suspect a bot problem can confirm it or rule it out in an afternoon.

Signal

Where to look

What it suggests

Response times rising without a matching rise in human sessions

Hosting dashboard or server monitoring, compared with analytics

Automated load is consuming capacity that visitors do not account for

Large request volumes from AI user agents

Raw access logs grouped by user agent

Specific crawlers are visiting heavily; check which pages they request

Crawlers requesting archive, filter or internal search URLs

Access logs filtered by URL pattern

Crawl is being spent on low-value pages that are probably uncached

Spikes in 5xx errors or HTTP 429 responses

Access logs, error logs, Search Console Crawl Stats

The server or a security rule is turning requests away, possibly including Googlebot

Average response time climbing in Crawl Stats

Google Search Console, Settings, Crawl stats

Google is seeing a slower server, which can reduce crawl capacity

Host status warnings

Crawl Stats host status in Search Console

Google has had trouble reaching or crawling the site reliably

Two practical notes. First, verify before you act. A request that says it is Googlebot or GPTBot may not be. Google documents a verification process for its crawlers, and OpenAI publishes the IP ranges its crawlers use, so you can check whether traffic claiming a famous name actually comes from that operator. Second, look at what bots request, not only how much. A crawler hammering filtered category pages is a URL-inventory problem as much as a traffic problem.

Building a practical AI-crawler policy

Decide by purpose, not by fear

Category

Typical purpose

Benefit to your site

Reasonable default

Search engine crawlers (for example Googlebot, Bingbot)

Index pages for search, including AI features in search

Rankings, citations and referral traffic

Allow; verify identity; never rate-limit into errors

AI search crawlers (for example OAI-SearchBot)

Build the index AI answers cite

Citations and some referral traffic

Allow if AI search visibility matters to you

AI training crawlers and tokens (for example GPTBot, Google-Extended)

Collect or use content for model training

Indirect and long-term at best

A business decision: allow, limit or disallow by policy

User-triggered fetchers (for example ChatGPT-User)

Fetch a page because a person asked

Often a real prospect behind the request

Generally allow; treat like a visitor

Unverified or impersonating bots

Scraping, probing, abuse

None

Rate-limit or block at the server or firewall

The training row is the one with no universal answer. A publisher whose content is the product may reasonably refuse training use. A service business that wants its expertise reflected in how AI tools describe its field may decide the opposite. What matters is that the decision is deliberate, written down, and reflected in both robots.txt and server configuration.

robots.txt is a request, not a lock

The Robots Exclusion Protocol, standardised by the IETF as RFC 9309, is explicit that robots.txt rules are not a form of access authorization. Well-behaved crawlers follow them; others ignore them. robots.txt is the right place to state your policy, but enforcement against bots that do not comply has to happen at the server, firewall or CDN layer.

A policy that keeps search visibility while declining training use might be structured like this:

User-agent: GPTBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: OAI-SearchBot

Allow: /

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Disallow: /?s=

Treat this as an illustration of structure, not a recommendation for every site. The right rules depend on your goals and your URL structure, and each crawler reads only the group that names it, falling back to the wildcard group when none does.

Rate-limit carefully, and never against verified search crawlers

Rate limiting is the most useful server-side control for aggressive bots, and also the easiest way to hurt yourself. Google names HTTP 429 as a signal that lowers its crawl capacity limit. A rule that throttles "anything crawling fast" can end up throttling Googlebot, which is precisely the crawler you most need to see a healthy server. Any rate-limiting or bot-management rule should exempt verified search engine crawlers and be checked against your logs after it goes live.

Make every request cheaper

The most durable fix for bot load is to reduce what each request costs, whoever sends it:

  • Page caching, so that most HTML requests never reach PHP or the database.
  • Conditional requests. Google recommends supporting HTTP 304 (Not Modified), which lets an unchanged page be revalidated without sending it again and saves bandwidth and server work.
  • Static asset caching with sensible cache headers, so crawlers and browsers do not keep refetching files that have not changed.
  • Compression and efficient images, which cut the bytes every visitor and every bot has to download.

Shrink the crawlable surface

Every low-value URL a crawler can reach is work your server does for nothing. Consolidating duplicates, returning 404 or 410 for content that is permanently gone, keeping internal search results out of the crawl, and controlling faceted navigation all reduce bot load while helping search engines focus on the pages you actually want indexed. Google's crawl budget guidance makes the same point from the indexing side: crawling wasted on unimportant URLs can leave less attention for the rest of the site. A technical SEO review is often the quickest way to find where that waste sits.

This is also where content quality and hosting meet. A large archive of outdated or thin pages does more than weaken a site's topical focus. It is crawl surface that has to be served, again and again, to every bot that walks it.

Questions to ask before you choose a host

Most of the controls above depend on what your hosting environment exposes. Before you choose, or renew with, a web hosting provider, it is worth asking questions that reflect what traffic actually looks like now.

Question

Why it matters

What a useful answer looks like

Can I access raw server access logs?

You cannot manage bot traffic you cannot see

Downloadable or live logs that include user agent, IP, URL, status code and response time

What happens when my account reaches CPU or process limits?

Determines whether bot bursts slow down human visitors or trigger errors

A clear description of limits, throttling behaviour and upgrade paths

Can I set rate limits or bot rules by user agent or IP range?

Lets you slow abusive bots without touching search crawlers

Configurable rules, ideally with verified-bot handling and logging

Do security or firewall features block crawlers by default?

Hosting-level blocks can override a permissive robots.txt

Defaults that are documented and that you can review and adjust

What page caching is available at the server level?

Caching is the biggest lever on the cost of each request

Server-side page caching that works with your CMS, with control over exclusions

How can I monitor time to first byte?

Response time feeds both user experience and crawl capacity

Tools or reports that show response-time trends, not just uptime

Where are the servers and backups physically located?

Affects latency for your audience and where your data resides

Specific data centre and backup locations, stated plainly

How does support help diagnose a traffic spike?

Bot incidents resolve faster with someone who can read the logs

Support that can identify top requesters and suggest rules

None of these questions is exotic. A provider that cannot answer them clearly, however, is asking you to manage today's traffic mix with yesterday's tools.

Shared, VPS or dedicated: matching the plan to machine traffic

No plan type makes bot traffic disappear, but different environments give you different room to absorb it and different control over it.

Shared hosting is cost-effective for small sites with modest traffic, especially when strong page caching is in place. Its limitation is that resources and server-level controls are shared, so you depend on the host's defaults for bot handling and on per-account limits that a crawl burst can reach.

A virtual private server gives you guaranteed resources and, depending on how it is managed, much more control over web server configuration, caching layers and rate-limiting rules. For a growing store, membership site or busy WordPress install, the combination of resource headroom on a VPS and configuration access is often where bot load stops being a recurring problem.

Dedicated servers suit sites with large catalogues, heavy dynamic content or compliance requirements that call for full control of the stack. Moving to dedicated hardware brings more power and more responsibility: someone has to configure and maintain it well.

Managed WordPress hosting can be a sensible middle path for teams without in-house server expertise, provided the managed layer includes good caching and sensible bot handling and still lets you see what is happening. In a managed WordPress environment, the questions above still apply. "Managed" should mean someone else does the work, not that you lose visibility into it.

The performance-versus-cost decision is no longer only about peak human traffic. Background machine traffic is now part of the baseline your plan has to carry every day.

The Canadian angle: latency, residency and the AI tools on your site

Distance still matters for people

Bots do not care how far away your server is. Your customers do. Light travels through optical fibre at roughly two-thirds of its speed in a vacuum, which works out to about one millisecond of round-trip delay for every hundred kilometres of cable, before routing, connection handshakes and processing are added. A site serving mostly Canadian visitors from a data centre on another continent starts every request at a disadvantage that no caching plugin fully removes. Hosting close to your audience keeps latency, and with it time to first byte, lower for the visitors who actually buy.

4GoodHosting operates from Canadian data centres in Vancouver and Toronto, placing infrastructure close to population centres on both sides of the country. Our page on Canadian data centre locations has the details.

The AI features you add send data somewhere else

There is a less obvious privacy dimension to AI becoming faster and cheaper. As AI features fall in price, more Canadian businesses add them to their own sites: chat assistants, AI-powered site search, automated support and content recommendations. Most of these work by sending what a visitor types, sometimes including names, order numbers or account details, to an external AI service for processing.

Hosting your website in Canada does not change where that third-party service processes the data. Under the accountability principle in PIPEDA, an organization remains responsible for personal information it transfers to a third party for processing. The Office of the Privacy Commissioner of Canada's guidance on cross-border processing expects organizations to protect that information through contractual or other means and to be transparent that it may be processed in another jurisdiction. Organizations subject to Quebec's private-sector privacy law face additional obligations before personal information is communicated outside Quebec.

The practical step is straightforward. Before adding an AI feature to a site, find out what data it sends, where that data is processed and retained, and whether your privacy notice and contracts reflect it. Combining Canadian hosting for your site and its databases with a clear understanding of any third-party AI processing gives you an accurate data-residency picture rather than an assumed one. Specific legal obligations depend on your circumstances, so confirm them with qualified counsel.

Three situations where this shows up

The scenarios below are illustrative composites, not client case studies.

A regional online retailer runs a WooCommerce store with several thousand products and filterable category pages. Response times climb every afternoon, yet analytics shows no matching rise in visitors. The logs reveal heavy crawling of filtered URLs by several AI and SEO crawlers. The fix combines keeping filter combinations out of the crawl, page caching for category views, and a rate rule for unverified bots, with verified search crawlers explicitly exempted. Checkout speed recovers without blocking any search engine.

A small digital agency hosts thirty client WordPress sites in one environment. A security plugin's "block bad bots" setting, switched on across every site, has been returning errors to a legitimate crawler it misidentified. Search Console for two client sites shows rising crawl errors. The agency replaces the blanket rule with a written AI-crawler policy for each client, verifies crawler identities before blocking anything, and moves its heaviest clients to a plan with dedicated resources.

A professional services firm wants an AI assistant on its website to answer common client questions. Before launch, the firm checks what the assistant sends to its vendor, updates its privacy notice, and keeps intake forms that collect sensitive details out of the assistant entirely. The website stays on Canadian infrastructure, and the AI processing is disclosed rather than assumed.

A 30-day plan for getting machine traffic under control

Week one: measure. Pull at least a month of access logs if your host keeps them. Group requests by user agent, list the URLs bots request most, and note status codes and response times. Compare the picture with the Crawl Stats report in Search Console. The goal is a factual baseline: who visits, what they take, and what it costs.

Week two: decide. Write a short AI-crawler policy. For each category in the table above, record whether you allow, limit or disallow it, and why. Name an owner for the policy, because the list of crawlers changes and someone needs to keep it current.

Week three: implement. Update robots.txt to state the policy. Add server or firewall rules for unverified and abusive bots, with verified search crawlers exempted. Confirm that page caching covers the URLs bots hit most, and that unchanged pages can answer with HTTP 304. Close off crawl traps such as internal search results and unbounded filter combinations.

Week four: verify and adjust. Check the logs for any errors served to Googlebot or other search crawlers. Watch average response time in Crawl Stats and on the hosting side. Compare human-visitor performance with your week-one baseline. Adjust any rule that is catching the wrong traffic.

After that, a quarterly review is usually enough: new crawlers appear, operators change how they identify themselves, and your own priorities about training use may shift.

Where this is heading

AI is moving from answering questions to taking actions: comparing quotes, checking availability, filling in forms on a user's behalf. That points to more user-triggered traffic and more agent sessions that look like visits but are not people. It also points to better tools for telling automated visitors apart. Google already documents an experimental method for authenticating its crawler requests with cryptographic signatures, and similar approaches are emerging elsewhere. Over time, verified identity should make it easier to welcome the bots that help you and refuse the ones that do not.

Two things are unlikely to change. Search engines and AI systems will keep favouring content they can fetch quickly and read cleanly. And every one of those fetches will keep landing on a server that someone has to pay for, monitor and tune. The sites that do well will be the ones whose hosting treats machine traffic as a normal part of the workload rather than an anomaly.

FAQ

Does AI crawler traffic slow down my website?

It can. Bots that request uncached pages such as archives, filtered product views and internal search results make the server generate each page from scratch. On hosting with fixed CPU or process limits, a burst of those requests can delay human visitors. Strong page caching and sensible bot rules usually keep the effect small.

Should I block AI crawlers?

Block by purpose rather than all at once. Crawlers that power search and AI search citations bring visibility and traffic, so most businesses allow them. Training crawlers are a business decision. Unverified or impersonating bots are reasonable to rate-limit or block at the server.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google states that Google-Extended controls use of content for training and grounding in its Gemini products and does not affect inclusion in Search, including AI Overviews and AI Mode. Those features rely on ordinary Googlebot crawling.

Does robots.txt stop all AI bots?

No. RFC 9309, the standard for robots.txt, states that its rules are not a form of access authorization. Compliant crawlers follow them; others do not. Enforcement against non-compliant bots happens at the server, firewall or CDN level.

Can bot traffic affect how Google crawls my site?

Indirectly, yes. Google adjusts its crawl capacity limit based on how quickly and reliably your server responds, including time to first byte, and lowers it when it sees server errors or HTTP 429 responses. If other bots slow your server down or your rate limits catch Googlebot, Google may crawl less.

How do I check whether a bot claiming to be Googlebot is genuine?

Use the verification methods the operator publishes. Google documents how to verify that a request comes from its crawlers, and OpenAI publishes the IP ranges its crawlers use. A user-agent string on its own proves nothing.

If my site is hosted in Canada, does data from an AI chatbot on it stay in Canada?

Not necessarily. The chatbot's AI processing happens wherever its provider runs it, which may be outside Canada. Check what data the tool sends and where it is processed, and reflect that in your privacy notice; confirm specific obligations with counsel.

What kind of hosting handles AI crawler traffic best?

There is no single answer. Small sites with good caching often do well on shared hosting. Larger or more dynamic sites usually benefit from a VPS or dedicated server, where resources are guaranteed and server-level rules can be configured. Whatever the plan, log access and caching control matter as much as raw capacity when comparing web hosting providers.

  1. Cheaper, faster AI has made automated visitors a majority of web traffic; Imperva's 2026 report puts automation at about 53% for 2025.
  2. AI traffic falls into three groups with different value to a site: training crawlers, AI search crawlers and user-triggered fetchers. Impersonators are a fourth problem.
  3. Uncached requests to deep pages are what make bot traffic expensive, and they can slow responses for real visitors.
  4. Google lowers its crawl capacity when response times rise or it receives errors and HTTP 429 responses, so unmanaged bot load and careless rate limits can reduce Google crawling.
  5. Google's AI features need no special treatment, but hosting and CDN settings must not block Googlebot; Google-Extended does not control AI Overviews.
  6. A written, purpose-based crawler policy, enforced at the server and checked against logs, works better than blocking everything or nothing.
  7. Canadian hosting keeps latency low for Canadian visitors, but AI features added to a site may process data elsewhere and need their own privacy review.

Conclusion

AI becoming faster is not just a story about chat tools. It is a change in who visits your website. A growing share of requests now comes from training crawlers, AI search crawlers and agents acting for people, and each of those requests lands on your server alongside real customers and search engines.

The sites that handle this well do a few unglamorous things consistently. They measure what is actually hitting the server. They decide, by purpose, which bots to welcome and which to limit. They make every request cheaper through caching and clean URL structures, and they make sure no hosting-level rule accidentally turns away the crawlers that drive visibility.

When you next compare web hosting providers, weigh log access, caching control, resource headroom and server location alongside price and storage. Those are the features that determine whether your site stays fast for people while it serves the machines that increasingly decide who gets found.

Want hosting where you can see who is hitting your server, tune caching for the pages that matter, and run your site from Canadian data centres? Explore 4GoodHosting's Canadian web hosting plans and talk to our team about the setup that fits your traffic.

Get in Touch

message
Your form has been submitted successfully.
We'll be in touch with you shortly.
Your email address will not be published. Fields marked with an asterisk (*) are mandatory.
+1 S
You may also like: