Last month a developer on r/webdev posted real Vercel logs from one site: Claude alone made 819,256 requests in 15 days. That is roughly 55,000 hits a day, every day, from a single AI crawler — and the Vercel bill went from $30 in November to $1,933.93 the next month. The same thread mentions Perplexity at 2.5M and Meta at 11.1M, but Claude was the new entry that surprised people, because most WordPress site owners did not even know its crawler had a name yet.
That crawler has a name. It is ClaudeBot, the user-agent Anthropic uses to fetch web content for training and product use. If you run WordPress and you have not opened your access log in a while, there is a very good chance ClaudeBot is already in there. The question every site owner asks me now is the same: do I allow it, do I block it, or is there a middle option?
TLDR: ClaudeBot is real, it hits WordPress sites hard, and the choice is not binary. You can allow it everywhere (full AI visibility, full server cost), block it everywhere in robots.txt (zero AI visibility, lower cost), or — the answer I push clients toward — allow it on the pages you want cited and block it on the pages that drain your server. What you will learn: how to identify ClaudeBot in your access log, how to allow or block it per-page in WordPress, how this differs from Claude-User and Claude-SearchBot, and how RankReady tracks every ClaudeBot hit on your site in real time so you can make that decision with data instead of guesswork.
What ClaudeBot actually is (and the other two Anthropic bots almost nobody mentions)
ClaudeBot is one of three crawlers Anthropic operates. The other two are Claude-User and Claude-SearchBot. They look similar in your access log, but they have completely different jobs, and that matters because the allow-or-block decision changes for each one.
| Bot | What it does | If you block it, you lose |
|---|---|---|
| ClaudeBot | Crawls the open web to collect training data for Claude models | Future model training on your content |
| Claude-User | Fetches a URL in real time when a Claude user pastes a link or asks Claude to read a page | The chance to be cited inside a live Claude conversation |
| Claude-SearchBot | Indexes pages to power Claude’s search-result quality (the Sonnet/Opus search feature) | Visibility in Claude’s search-grounded answers |
Most WordPress posts you will find online conflate all three under “ClaudeBot.” They are not the same. Blocking ClaudeBot stops training data collection but does not stop Claude-User or Claude-SearchBot. That distinction is the difference between “I do not want my content used to train future models” and “I do not want to appear in Claude answers at all.” Pick the right bot for the right outcome.
The user-agent strings (per Anthropic’s official documentation) all start with Mozilla/5.0 (compatible; and then differ at the end. ClaudeBot ends with ClaudeBot/1.0; [email protected]. Claude-User and Claude-SearchBot follow the same pattern with their own names. The IP range list is published and refreshed at https://claude.com/crawling/bots.json — at time of writing it is a mix of 216.73.216.0/22 and around twenty 34.x.x.x Google Cloud /32 addresses. If a request claims to be ClaudeBot but the source IP is not in that JSON, it is a spoofer. Treat it as such.
How to spot ClaudeBot in your WordPress access log right now
You do not need a plugin to confirm ClaudeBot is hitting you. SSH into your host and run this one line against your nginx or Apache access log:
tail -10000 /var/log/nginx/access.log | grep -i ClaudeBotOr on Apache:
tail -10000 /var/log/apache2/access.log | grep -i ClaudeBotIf you see output, ClaudeBot is already crawling you. On a typical mid-traffic WordPress site I checked last week, ClaudeBot was hitting roughly one request every four seconds — not enough to crash the site, but more than enough to fill PHP workers if you are on a cheap shared host with no page cache for bot traffic. Here is what a clean log slice from a real WordPress install looks like:
If you are on managed WordPress hosting (Kinsta, WP Engine, Cloudways, Pantheon, Pressable) and you do not have shell access, the access log is still there — every panel has a “view raw access log” option. Use the host search and filter for ClaudeBot. Same answer in fifteen seconds.
What other WordPress site owners are actually doing about it
I pulled the last twelve months of r/Wordpress and r/webdev threads via the Reddit JSON API to see what people who run real sites are saying. Two things stood out. First, the Vercel post hit 3,200 upvotes in a few weeks — this is not a quiet topic. Second, the WordPress-specific complaint is consistent: AI crawlers hit pages that are not in the page cache, so they hit PHP every time, and on WordPress that means database queries, theme rendering, plugin filters — the works.
A WPMU DEV support agent said it plainly in the r/Wordpress “Is ClaudeBot ignoring crawl limits?” thread: “We see it happening daily and it will cause spikes in the CPU usage especially when hitting a search query such as product, native /?s= or event calendars.” Another user in the same thread, thedragonturtle, made the cache point cleanly: “The AI crawlers don’t use your site like humans, so whatever pages they hit are rarely in the cache so they create MORE server load than real users.”
That is the WordPress-specific bit nobody explains well. On a static site, a crawler hit costs you almost nothing. On WordPress, every uncached ClaudeBot hit can trigger a full PHP execution, twenty plugin filters, and a handful of database queries. Multiply that by 55,000 requests a day and you have a real bill — even if it is just the host pushing you into the next plan tier.
The honest allow-or-block framework
I have seen people argue both extremes online. Block everything because the deal is broken (no referral traffic back). Allow everything because the future of search is here. Both are too clean. Here is the framework I actually run with for POSIMYTH and for client sites.
Allow ClaudeBot if:
- Your business benefits from being cited by AI tools (most B2B SaaS, agencies, publishers, course creators)
- You have working page caching that absorbs bot hits (Cloudflare APO, WP Rocket, FastCGI cache, LiteSpeed)
- Your content is informational, evergreen, and not paywalled
- You have a way to track which pages get cited (see RankReady below)
Block ClaudeBot if:
- Your content is your monetised product (premium courses, paid research, gated PDFs)
- You are on a shared host where PHP workers cap out at 4-8 and bot traffic measurably slows real users
- You have measured no AI referral traffic over the last 90 days
- You explicitly do not want your work used as training data
Do both (the middle path):
- Allow ClaudeBot on the cacheable, evergreen pages you want cited
- Block ClaudeBot on
/checkout/,/my-account/,/?s=,/cart/,/wp-admin/,/wp-json/and any high-cost dynamic URL - Allow Claude-User and Claude-SearchBot everywhere unless you have a specific reason not to (these are the bots that produce the citation, not the training crawl)
That last option is the one I push clients toward. It costs you almost nothing in citation visibility and saves the server from the worst of the burn.
The robots.txt rules that actually work (copy-paste)
This is the block I drop into the robots.txt for sites taking the middle path. It allows ClaudeBot on content but blocks the expensive dynamic stuff:
# Allow ClaudeBot on content, block on dynamic/admin paths
User-agent: ClaudeBot
Crawl-delay: 2
Disallow: /wp-admin/
Disallow: /wp-json/
Disallow: /checkout/
Disallow: /cart/
Disallow: /my-account/
Disallow: /?s=
Disallow: /*?add-to-cart=
Disallow: /*?orderby=
# Same rules for Claude-SearchBot
User-agent: Claude-SearchBot
Crawl-delay: 2
Disallow: /wp-admin/
Disallow: /wp-json/
Disallow: /checkout/
Disallow: /cart/
Disallow: /my-account/
Disallow: /?s=
# Always allow Claude-User (the live citation bot)
User-agent: Claude-User
Allow: /If you want to block ClaudeBot entirely (no training, but still let citation bots through):
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /If you want to block everything Anthropic:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /Anthropic states in their official docs that all three bots respect robots.txt and that Crawl-delay is honoured “where appropriate.” The r/Wordpress thread I quoted earlier had one user (Dr_Beatdown) reporting sustained high request rates from ClaudeBot until they blocked it at the .htaccess level — so if you do block it and the traffic does not drop within 24 hours, escalate to host-level rules. That is rare in my experience, but it does happen.
A sister POSIMYTH post — the WordPress robots.txt for AI crawlers deep dive — walks through editing robots.txt on every major host (WP Engine, Kinsta, Cloudways) including the Yoast/Rank Math virtual robots.txt override. Read that one alongside this if you have never edited robots.txt on a managed host before.
What llms.txt adds on top of robots.txt
There is a newer file format being adopted by AI labs: llms.txt. It does not block anything — it tells AI crawlers which of your pages you want them to use, and in what order. ClaudeBot does not yet officially read llms.txt, but Anthropic’s own documentation publishes one for their site, which is a strong signal. I cover the WordPress setup in detail in the llms.txt for WordPress post. Treat it as the carrot and robots.txt as the stick.
Two newer pieces of the AI-WordPress stack worth knowing about while you are here: WebMCP (the protocol that lets AI agents read your site’s structured data live) and MCP servers in general. The WebMCP for WordPress and What is an MCP Server on WordPress posts dig into those. ClaudeBot is the crawl side. WebMCP is the live-query side. Both matter.
The Elementor angle — block ClaudeBot from your builder previews
If you use Elementor, there is a specific path you almost certainly want to disallow: /?elementor-preview=. Every page on your site has a builder-preview URL that renders the full builder UI. ClaudeBot does not need it, it triggers heavy backend execution, and it can leak draft content into training data. Add this to the block above:
User-agent: ClaudeBot
Disallow: /?elementor-preview=
Disallow: /?elementor_library=Same logic for Gutenberg’s /?wp-block-editor= preview routes if you use them.
If you build custom headers, footers, or page templates in Elementor, The Plus Addons Header Builder makes one more decision easy: which header variants you serve to logged-out visitors (the bot path) versus logged-in visitors. Most sites do not need to differentiate, but if you do, it is a one-click setting.
Where ClaudeBot fits in the bigger AI-WordPress picture
The trap to avoid: thinking ClaudeBot is the whole AI traffic story. It is one bot. There are dozens. GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Bytespider, Amazonbot, Meta-ExternalAgent, and a long tail of less-known crawlers. Each has its own user-agent, its own IP range, its own robots.txt opinion. The Vercel post that kicked off this article showed Meta sending 11 million requests in 30 days to a single site — almost half of real user traffic. Claude was just the new entrant.
Managing all of that by hand in robots.txt is doable. Watching all of that in your access log every week, finding the pages they over-crawl, deciding which to allow, which to block, which to rate-limit, which produced an actual AI citation last month — that is the bit that breaks down at scale.
RankReady’s AI Crawler Log — the dashboard view of every ClaudeBot hit
This is the part of the post where I tell you what we built and why it matters for the ClaudeBot decision specifically.
RankReady is a WordPress plugin we built at POSIMYTH for AI-first SEO. One of its core modules is the AI Crawler Log, and it solves the exact problem this post describes. Four ways the log changes how you decide on ClaudeBot:
1. Real-time hit feed per bot
RankReady sits at the WordPress request layer (before page cache) and logs every AI crawler hit with bot name, IP, URL, response code, response size, and timestamp. You do not need to SSH or grep. Open the dashboard, filter to ClaudeBot, see the last 24 hours.
2. Per-page hit count and citation tracking
The log groups hits by URL. You see exactly which posts ClaudeBot is hammering — and which ones it ignores. Pair that with the citation tracker module and you see which pages produced an actual Claude citation in the last 30 days, so you can stop guessing what to allow.
3. One-click block per bot, per URL pattern
Instead of editing robots.txt by hand, you toggle ClaudeBot off for /checkout/, /cart/, /?s= and the like inside the plugin. RankReady writes the rule to the WordPress virtual robots.txt (or appends to a real one if you have it), so it survives theme swaps and host migrations.
4. Spoof detection — verify the bot is real
RankReady checks every claimed ClaudeBot request against the published claude.com/crawling/bots.json IP list. Spoofers — and there are plenty — get flagged automatically. You stop wasting block rules on traffic that was never really ClaudeBot.
Full module list in the current RankReady build:
- AI Crawler Log (the one this post is about)
- AI Citation Tracker (Claude / ChatGPT / Perplexity / Google AI Overview)
- AI-Optimised Schema Generator
- llms.txt Builder
- robots.txt Manager (per-bot, per-URL rules)
- FAQ Schema for AI Search
- Internal Linking for AI Discovery
- Content Freshness Signals
- WebMCP Endpoint
Your 10-minute ClaudeBot action checklist
- SSH or open your host’s raw access log. Grep for
ClaudeBot. Confirm it is hitting you. - Decide which of the three paths you want: allow everywhere, block everywhere, middle path.
- Copy the relevant robots.txt block from the section above and paste it into your site’s
robots.txt(or the Yoast/Rank Math virtualrobots.txteditor). - Add the Elementor preview disallow if you use Elementor.
- Wait 24 hours, re-check the access log. ClaudeBot should be hitting only the URLs you allowed.
- Cross-check the requesting IPs against
https://claude.com/crawling/bots.jsonto filter spoofers. - Install RankReady if you want this without doing it by hand and across every other AI bot at the same time.
Wrapping up
ClaudeBot is not the enemy and it is not free traffic. It is a crawler with a clear identity, a public IP list, and robots.txt compliance — which puts it ahead of most of the AI bots hitting your site right now. The right move depends on your business: allow it where citations help you, block it where dynamic pages drain your server, and verify every hit is genuine before you waste a rule on a spoofer.
Whatever you choose, see the data before you decide. A grep of last week’s access log is the floor. RankReady’s AI Crawler Log is the version that actually scales when you have ten of these bots to manage at once. And if you build with Elementor, The Plus Addons keeps the front-end speed up so the bot hits that do come through cost you less PHP per request.
