Artificial Intelligence Web Scrapers Dominate AI Robot Web Traffic
In a recent report, Fastly, Inc. has shed light on the increasing presence of AI crawlers in online traffic, revealing a complex challenge to security, performance, and operational resilience.
According to the Q2 2025 Threat Insights Report, automated bot traffic already accounts for 37% of observed activity across Fastly's global network. This trend is particularly pronounced in North America, where nearly 90% of observed AI crawler traffic originates.
The report underscores the need for more transparent bot verification, clearer signalling by bot operators, and smarter bot management strategies. This is crucial to mitigate growing exposure to shadowy automation, attribution gaps, and escalating infrastructure costs.
One of the key findings is the geographic bias in LLM training datasets, with every other region, including Europe, Asia, and Latin America, seeing a significantly smaller share of AI crawler traffic. This trend is particularly prevalent in the commerce, media & entertainment, and high-tech sectors, which face the highest levels of scraping for training AI models.
The report also highlights the impact of specific bots, such as fetcher bots, including those used by ChatGPT and Perplexity. These bots are driving high real-time request volumes, sometimes exceeding 39,000 requests per minute. Notably, ChatGPT generates the most real-time traffic to websites, with 98% of fetcher bot requests attributable to OpenAI's bots.
Fastly's findings also reveal the dominance of certain players in the AI crawler landscape. Meta's AI bots alone generate 52% of AI crawler traffic, more than double that of Google (23%) or OpenAI (20%). However, there are no search results providing information on the largest producers of AI crawler traffic in Q2 2025 or their respective shares.
The surge in fetcher request volume is putting pressure on unprotected origin infrastructure, consuming bandwidth, overwhelming servers, and mimicking the effects of DDoS attacks. This underscores the importance of robust bot management strategies to protect online assets and ensure optimal performance.
Fastly's insights offer a detailed view of how AI-driven automation is reshaping online traffic, with visibility into over 130,000 applications and APIs across various industries. However, the lack of bot verification persists, making it difficult for security teams to distinguish between legitimate automation and malicious impersonation.
In conclusion, Fastly's Q2 2025 Threat Insights Report provides a comprehensive overview of the growing influence of AI crawlers and the challenges they pose. As AI continues to evolve, it is crucial for organisations to adopt transparent bot verification, clearer signalling by bot operators, and smarter bot management strategies to ensure a secure and efficient digital landscape.
