How the AI Crawler GPTBot Crawled a 60,000-Page Website

Metihan Eshilyurt запустил an AI-generated website with 60,000 pages for $10 and began monitoring bot visits.

The logs show the current level of the OpenAI crawler.

In the first 12 hours, GPTBot made over 29,000 requests.

Googlebot — 11.

For context: this is about 470 times more aggressive crawling intensity on a fresh domain with zero backlinks, zero social signals, and no submission to the console.

GPTBot found the site through the XML sitemap and simply started consuming everything at a rate of about 1 request per second.

The data reveals what early infrastructure looks like before optimization mechanisms are implemented.

Google spent 25 years solving the problem of crawling budget distribution — understanding which pages deserve attention, how often to revisit them, and how to do all this without unnecessarily DDoSing servers.

For them, this is a long-closed question.

OpenAI does not have this yet.

The behavior of the crawler that Metihan observed is a result of having computational power and ambition, but not yet having built the layer of efficiency that comes from working at Google’s scale for decades.

This resembles early advertising auction systems, where first-generation bidders simply flooded the problem with resources, and complexity came later when the economy forced changes.

GPTBot right now is equivalent to a bidder with an unlimited budget and no bid cap; effective in reach but terrible in optimization.

The main practical point: if you haven’t explicitly blocked GPTBot in robots.txt, it crawls your site just the same.

Most website owners are not even aware of this because client analytics miss it all.

Deploy server-side tracking with user agent parsing to simply see what is happening.

https://metehan.ai/blog/i-built-a-60-000-page-ai-website-for-10-gptbot-crawled-it-30-000-times-in-12-hours/