Free tool
AI Crawler Log Analyzer
See which AI crawlers hit your site, the paths they fetch, the status codes they get and robots.txt requests.
How it works
What is the AI Crawler Log Analyzer?
The AI Crawler Log Analyzer is a free tool that reads pasted Apache or nginx access log lines and reports only on AI crawlers. It shows hits per bot, grouped as training, search and retrieval, or user-triggered, the paths each one fetches most, the status codes it received, and whether it requested /robots.txt or /llms.txt.
How to use the AI Crawler Log Analyzer
- Copy lines from your Apache or nginx access log.
- Paste them into the box.
- Read the totals for training, search and user-triggered crawlers.
- Open each crawler's card for its status codes, top paths and robots.txt and llms.txt requests.
How do I know if AI bots are crawling my site?
Your server access log is the only place that shows it. AI crawlers do not run analytics scripts, so they are invisible in tools like Google Analytics. Each request they make leaves a log line with a user-agent string containing a token such as GPTBot or ClaudeBot.
Paste those lines here and the tool counts requests per token. If the result is zero, either no AI crawler visited in that period or a firewall or CDN blocked them before the request reached the server that wrote the log.
Which AI crawlers does the analyzer recognise?
Twelve user-agent tokens. Training: GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot and Meta-ExternalAgent. Search and retrieval: OAI-SearchBot, Claude-SearchBot and PerplexityBot. User-triggered: ChatGPT-User, Claude-User and Perplexity-User.
Google-Extended and Applebot-Extended are missing on purpose. They are robots.txt control tokens, not crawlers, so no request ever carries those names. Crawlers outside this list are not counted; the Server Log Analyzer shows every user agent.
What do training, search and user-triggered hits mean?
Training hits are pages collected for possible use in model training. Search and retrieval hits build the index an AI search product draws its answers and citations from. User-triggered hits happen when a person asks an assistant about a page or pastes its URL, so each is a sign that someone was shown or asked about your content.
The mix tells you how AI products use your site. Plenty of training hits and no search or user hits means your content is being collected but is not yet surfacing in answers.
Why check status codes for AI crawlers?
Because a crawler can only use what it receives. 200 means it got the page. 403 often means a firewall or bot rule is blocking it, perhaps without you knowing. 404 means it is following dead links. 5xx means your server failed under its requests.
The tool also reports whether each bot requested /robots.txt, which well-behaved crawlers do before fetching pages. If a bot ignores your rules, robots.txt will not stop it; enforce the block at the edge with the Cloudflare Rules Generator.
Do AI crawlers request llms.txt?
/llms.txt is a proposed convention for giving language models a summary of a site. It is not a standard and support varies, so your own log is the direct evidence: the tool shows, per crawler, whether a request for /llms.txt appears in the lines you pasted.
If you want to offer one, the llms.txt Generator writes it. To change which AI bots may crawl at all, use the AI Crawler Robots.txt Generator. Remember that user-agent strings can be spoofed and this tool does not verify IP addresses.
Questions
- Can a request pretend to be GPTBot?
- Yes. Anyone can send any user-agent string. Several operators, including OpenAI, publish IP ranges you can check a suspicious hit against. This tool does not do that check.
- Why does Google-Extended not appear in my logs?
- It is a robots.txt control token, not a crawler. Google's normal crawlers fetch the page and the token only governs how the content may be used.
- Which log format do I need?
- Apache or nginx combined format, which includes the user agent. Lines without a user agent cannot be matched to a crawler.
- Does more AI crawling mean more AI citations?
- Not necessarily. Crawling shows access, not use. To see whether assistants mention your brand, run the AI Brand Visibility Checker.
- Is the AI Crawler Log Analyzer free?
- Yes, with a free spyc.ai account. It runs in your browser, so nothing you type or paste is sent to a server.
Technical SEO Tools
More in Technical SEO Tools
Opengraph Viewer
Paste a URL and see the card it produces when someone shares it — title, description, image.
View OpengraphOpen Graph Generator
Write the Open Graph tags, preview the card, copy the markup.
Generate Open Graph tagsRobots.txt Checker
Fetch any site's robots.txt and test one URL against it: allowed, or blocked by which rule.
View robots.txtReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.