AI bot traffic: how to find AI crawlers in your server logs

Find AI crawlers in logs

To find AI crawlers, filter your raw access logs by known AI user-agent strings and count the hits for each bot. Then confirm the IP addresses are real. Use the server log, which records every request, to track crawlers that bypass the tracking scripts of JavaScript-based analytics tools such as Google Analytics.

Before you start, gather these:

• Read access to your web server's access logs, or to an export from your host
• At least 7 days of logs, since some bots visit in bursts
• A terminal with `grep` and `awk`, or a log viewer such as GoAccess
• The current AI user-agent list from the table further down
• The IP range files that OpenAI, Perplexity and other vendors publish

1. Locate the access log

Nginx writes to `/var/log/nginx/access.log` by default. On Debian and Ubuntu, Apache writes to `/var/log/apache2/access.log`. Red Hat systems use `/var/log/httpd/access_log`. Older days get rotated into numbered files such as `access.log.2.gz`, and managed hosts let you download the log from the control panel instead. If a load balancer sits in front of your server, pull its logs too, because the origin only sees the balancer's IP.

2. Filter by AI user-agent

Use a case-insensitive search on the bot names:

zgrep -hiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Meta-ExternalAgent" /var/log/nginx/access.log* > ai-hits.log

`zgrep` reads both compressed and plain files, so the rotated logs come along in one pass.

3. Count hits per bot

grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot" ai-hits.log | sort | uniq -c | sort -rn

The output ranks AI agents by their request counts. To see which pages each bot requests, swap the first command for `awk '{print $7}'` and run it on a file filtered to one bot.

4. Verify the IP addresses

Anyone can type "GPTBot" into a user-agent header, so a name match alone proves nothing. OpenAI and Perplexity both publish their crawler IP ranges as JSON files on their documentation pages. Check every IP in `ai-hits.log` against those ranges. For bots that run through a search engine, like Googlebot and Bingbot, do a reverse DNS lookup and then a forward lookup to confirm the hostname. Any hit that fails the check is a spoofed request and doesn't count as AI bot traffic.

What AI bot traffic reveals

AI bot traffic tells you which AI companies are reading your content and which pages they read. It also shows whether assistants are pulling your pages live to answer someone's question. This log data is the nearest thing you have to a signal from inside assistants such as ChatGPT and Claude. The same applies to Perplexity, and these signals are absent from your analytics.

A training crawler hitting your blog shows your text is a candidate for use in a future large language model. A hit from a search-index crawler means your page is a candidate for AI search results. Fetches from user-triggered agents mean a real person asked a question, and the assistant judged your page relevant to the answer at that moment.

An assistant can reuse content it has already indexed, so crawl activity alone cannot establish whether your pages appear in answers. On a small site, the counts can be low enough that a single busy week looks like a trend. Before you draw a conclusion, compare four weeks or more.

Reading server log entries

Each line in an access log records one request, and in the default "combined" format the user-agent is the last quoted field. Here's a GPTBot request, using a documentation IP:

192.0.2.44 - - [12/Mar/2025:08:14:02 +0000] "GET /pricing HTTP/1.1" 200 18342 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"

Reading left to right:

• `192.0.2.44` is the client IP, which you verify against vendor ranges
• The bracketed timestamp tells you when the bot came
• `GET /pricing` is the page it asked for
• `200` is the status code your server returned
• `18342` is the response size in bytes
• `"-"` is the referrer, which is empty because bots rarely send one
• The final quoted string is the user-agent, where the bot names itself

Formats differ from server to server. Some hosts log in JSON, and some administrators change the field order. Behind a proxy or CDN, the first field holds the proxy's address, with the real client IP in an `X-Forwarded-For` field. Before you trust any count, inspect 5 lines in the log to confirm which position holds the IP and which holds the user-agent. The `awk` field numbers in the commands above assume the default layout.

CDN analytics for AI agents

When your CDN analytics show AI bot visits, those are requests the CDN tagged as AI crawlers at its edge servers. The count covers pages served from cache without a request to your origin. The CDN figure is therefore higher than the one in your server logs, and for cached sites it's closer to the true total.

Cloudflare's dashboard has an AI crawler view (AI Crawl Control, formerly AI Audit) that breaks requests down by crawler and operator. It also shows which bots respect your robots.txt rules. Akamai and Fastly include bot categories in their bot management products. Each CDN uses its own detection, which combines user-agent and IP data with signals from request behavior. Its labels differ from your `grep` results, and requests your filter missed are still classed as AI traffic there.

Read the CDN figures alongside your logs as a second view of the same traffic. If the CDN shows heavy GPTBot traffic but your origin log shows almost none, caching explains the gap and nothing is broken. If the two disagree about which bots appear at all, check whether the CDN is blocking a crawler by default. Some CDN plans now block AI training crawlers unless you opt out.

Key AI crawler user-agents

Each major AI vendor uses separate user-agents for training and search indexing, with another user-agent for live user requests. The table maps the main ones. Vendors rename and add bots without much notice, so check their documentation pages every quarter.

Google-Extended and Applebot-Extended are 2 important names used only as robots.txt tokens. Google crawls with the normal Googlebot and Apple with Applebot, and the token just controls whether the content can be used for AI training. Microsoft Copilot gets its web results through Bingbot. For these 3 assistants, your logs can show the crawl but can't tell you the AI use.

Separating crawlers from assistants

To separate crawlers from assistants, sort every AI hit into one of 2 groups: bots that crawl on a schedule and agents that fetch a page because a person asked something. The user-agent tells you which group a hit belongs to. Names ending in "-User" identify live fetches, while "Bot" names crawl in the background.

The split is worth making because the 2 groups mean different things. Scheduled crawls go through hundreds of URLs in a session around the clock. A live fetch asks for 1 to 3 pages within seconds, and the timing follows human hours in your audience's time zones. So a spike in ChatGPT-User hits on a product page is a sign of real demand.

Robots.txt also works differently for each group. OpenAI and Perplexity both describe their user-triggered agents as acting for a person, and their documentation describes different treatment of robots.txt rules for these fetches and crawlers. If you block GPTBot, ChatGPT-User can still reach your pages.

The third group consists of browser-based agents, such as OpenAI's agent mode, which run a full browser and send a standard Chrome user-agent, so your log records them as normal visitors. The one partial signal is people who click through from an answer. Their visits carry a referrer such as `chatgpt.com` or `perplexity.ai`, and ChatGPT adds `utm_source=chatgpt.com` to the links it cites.

Metrics worth tracking

Track how much each AI bot crawls and which pages it requests to guide your decisions. Assess your server's responses to those requests and whether the crawls lead to visits. Raw request totals on their own tell you little.

• Requests per bot per week. Use this as the baseline. A sudden drop for one bot often points to a block you didn't intend.
• Unique URLs per bot. This shows coverage. A bot that requests 40 URLs 500 times each is not reading your whole site.
• Share of priority pages crawled. List your 20 to 50 most important URLs and check how many each bot reached in the period.
• Status code mix. Track the share of 4xx and 5xx responses for each bot. Errors waste crawls and keep pages out of AI indexes.
• User-triggered fetches per URL. Count ChatGPT-User and Claude-User hits by page, then do the same for Perplexity-User. This is the closest log signal to "an assistant used this page in an answer."
• AI referral visits. Count human visits with AI referrers or UTM tags in your analytics.
• Crawl-to-referral ratio. Divide a vendor's crawler requests by the visits that vendor sends you. It shows what you get back for the bandwidth.
• Bandwidth per bot. Sum the bytes field. Heavy crawlers can add real hosting cost on large sites.

Start with the first 4 metrics if your time is limited. They come straight from the log and need no join with analytics. Put the others in place once you have a month of baseline data.

Turning data into actions

Use the log findings to repair errors and add links to pages bots missed. Base your decisions to allow or block bots on those findings too. In my experience, the first audit turns up at least one crawler hitting broken URLs, and that's the easiest fix to make.

Start with errors. Find the URLs that return 404 or 5xx to AI bots, then redirect or repair them. Next, take the priority pages no AI bot has requested and check that they sit in your XML sitemap and get internal links from pages bots already visit. Then fetch those pages with `curl` and read the raw HTML. If the main text only appears after JavaScript runs, a crawler that doesn't render scripts sees an empty shell.

Access decisions come after that, and they depend on your business. A publisher that licenses content may block training crawlers while it allows search and user agents. A software company that wants to be cited allows all 3 groups. Write separate robots.txt rules for each user-agent token, and back them with firewall rules for bots that ignore robots.txt. If a single crawler overloads your server, rate-limit it by IP range to keep its access within manageable levels.

After every change, rerun the counts a week later. If they drop where you expected and stay level elsewhere, the change worked.

Checking results and fixing common errors

The audit has worked when each named bot has a verified hit count and your origin numbers make sense next to your CDN figures. If you see zero AI hits on a public site that's been live for months, something is wrong with the setup.

These are the usual causes and their fixes:

• No hits at all. You're reading the wrong log, or the CDN serves every page from cache. Check the CDN dashboard, or log at the edge.
• Every IP is the same. You're seeing the proxy's address. Log `X-Forwarded-For`, or switch on your CDN's real-IP module.
• Only today's data. Older entries sit in rotated `.gz` files. Use `zgrep` with a wildcard on the file name.
• Bots missing from the count. The filter is case-sensitive or out of date. Add `-i` and update the list of names.
• Verified share is low. Spoofed bots are common. Exclude them from every metric and think about blocking them at the firewall.

What to do next

Set up the weekly count from step 3 as a scheduled job and save its output, because one snapshot can't show whether AI bot traffic is growing or which pages assistants come back to. A month of history makes the user-triggered fetch counts useful. These show the pages that AI assistants rely on, and those are the pages to keep accurate and fast. Snoika audits websites and tracks brand visibility in Google and AI assistants to show what the assistants say about your brand. Check your AI visibility free.