AI crawlers now match Googlebot in traffic to our site
An analysis of 30 days of traffic from a site that hit the Hacker News front page shows AI crawlers (e.g., GPTBot, ClaudeBot) tied with search engine crawlers at 35% each. The article compares bot behavior, including most aggressive, polite, obsessive, and thorough crawlers.
What traffic & bots flow to your site after you make it to the front page of Hacker News? Do AI bots crawl as much as Google? Which is the most aggressive bot? Which is the least and thriftiest?
A recent post hit the front page on hacker news. Google just made you a search quality rater.You won't get paid.
This was something I wished to know so I built a little bot detector on this site. Of course using Claude, the bot.
The code basically measured the traffic over 30 days and I built a simple analyser within the code to output the following data.
Across 240,060 visits from 24 distinct bots, AI crawlers (35.0% of traffic) tied search engines (35.0%) for the top slot.
If you care about this industry then that's the headline. The rest is nerding in data.
The setup
We log every visit from a recognised bot user-agent into a single table. The fields are bot name, category, path, day, hit count. 240,060 hits, 24 bots, 5,074 distinct URLs touched.
We bucket bots into five categories: search (Google, Bing, DuckDuckGo, etc.), ai (GPTBot, ClaudeBot, PerplexityBot, AmazonBot, etc.), seo (Ahrefs, Semrush, Majestic, Screaming Frog), social (LinkedIn, Facebook, Twitter, Pinterest), and archive (Internet Archive).
Category split
AI crawlers tied search engines at 35.0% each.
Category
Hits
Share
AI crawlers
84,080
35.0%
Search engines
83,980
35.0%
SEO tools
50,820
21.2%
Social previews
20,700
8.6%
Archive
480
0.2%
Most aggressive bots
Hits per day, averaged over the period.
Bot
Category
Hits/day
Total hits
Unique paths
Googlebot
search
1,228
36,840
1,474
Bingbot
search
1,187
35,610
1,785
AmazonBot
ai
1,101
33,040
1,442
MajesticBot
seo
1,062
31,860
2,026
ChatGPT-User
ai
812
24,350
239
ClaudeBot
ai
581
17,430
647
AhrefsBot
seo
485
14,550
1,279
LinkedInBot
social
478
14,340
15
Most polite bots
Hits per path. Lower = visit and leave.
Bot
Category
Hits/path
Unique paths
AhrefsBot
seo
11.4
1,279
Applebot
search
11.9
575
MajesticBot
seo
15.7
2,026
SemrushBot
seo
16.2
265
Bingbot
search
20.0
1,785
AmazonBot
ai
22.9
1,442
Googlebot
search
25.0
1,474
ClaudeBot
ai
26.9
647
Most obsessive bots
Re-scrapers. Few URLs, hammered repeatedly.
Bot
Hits/path
Total hits
Distinct paths
LinkedInBot
956.0
14,340
15
FacebookBot
170.3
6,130
36
ChatGPT-User
101.9
24,350
239
Internet Archive
16.0
480
30
Most thorough explorer
Distinct URLs touched.
Bot
Distinct paths visited
Total hits
MajesticBot
2,026
31,860
Bingbot
1,785
35,610
Googlebot
1,474
36,840
AmazonBot
1,442
33,040
AhrefsBot
1,279
14,550
ClaudeBot
647
17,430
Applebot
575
6,840
Rare visitors
Bots that barely show up across 30 days.
Bot
Total hits
Days active
Internet Archive
480
2
Baiduspider
480
9
DuckDuckBot
470
11
Twitterbot
160
4
DotBot
80
5
Pinterestbot
70
4
Google-Extended (Gemini)
60
1
Screaming Frog
50
5
Methodology
Source: nexus_bot_visits table on mojodojo.io's production database. One row per (bot, path, day) with a hit counter.
Bot detection: User-Agent substring match against a curated list of 32 known bot patterns. Unknown / spoofed UAs are excluded.
Filed under
ai-visibilityseoanalyticsbots
Written by
Ajay C
Ajay is the CTO of Mojo Dojo with over 15 years of experience in digital marketing.
Keep reading
More from Insights
Google just made you a search quality rater. You won't get paid.
TL;DR AI is killing the click. Death of the click kills the incentive for links. This leaves Google...
7 June 2026
Meta is selling organic reach for a flat fee.
For fifteen years, agencies sold clients a dream: "earned media" on someone else's property. We...
7 June 2026
How to write a great value proposition? (Template included)
What is value proposition? A great value proposition makes your business the obvious choice amongst...
6 June 2026
View all articles