AI crawler hits on our own site, 12-25 September 2026
Almost nobody can tell you what AI crawlers actually asked their site for, because only the site owner holds the log. We do hold it, and we are publishing the count, including the requests that failed.
The numbers
| Value | What it counts | Source / definition |
|---|---|---|
| 103,222 | request lines parsed | All lines in 26 log files covering 2026-09-12 11:28 to 2026-09-25 16:10 (Asia/Shanghai). Source: this site's nginx logs. |
| 1,722 | hits from AI crawlers | Requests whose user agent matches one of the AI crawler names we track, over the same window. Definition and per-agent breakdown are in the published dataset. |
| 1,047 | AI crawler hits answered with 404 | The requested URL did not exist. That is 61% of all AI crawler hits in the window. |
| 673 | AI crawler hits answered with a real page | HTTP 200 responses to AI crawler requests, same window. Two hits were redirects. |
| 10 | AI crawler user agents seen in the window | Distinct AI crawler names in our logs; classic search crawlers such as Googlebot and Bingbot are counted separately and are excluded from this number. |
What one record is
One hit is one request line in our nginx access log whose user agent matches a known AI answer-engine crawler. Hits are counted whether or not the URL existed.
Records
Most guidance about AI crawlers is written from the outside, using whatever a crawler reports about itself. This page uses the inside view: our own server logs, counted by our own script, with the failures left in. Every figure in this section, including the per-day counts, is in the published dataset.
- Two days carry most of the traffic. 682 hits on 12 September and 362 on 13 September, then almost nothing until 25 September, when GPTBot alone asked for 436 URLs. A one-off sweep and a steady re-crawl look identical from the outside and completely different in a log.
- The missing URLs are not readers. The most-requested missing paths are credential and configuration probes: cloud credential files, repository metadata, an internal metadata address reached through a proxy parameter, a graph console. That is scanner traffic wearing a crawler name. It is counted as crawler traffic here because we count what the log says, not what we would prefer.
- What did get read. Where a crawler found a real page it usually took the front page or a dataset file. The crawlers that arrived on 12 and 13 September mostly collected errors, and most of that group has not returned since.
- Why we publish it. If an answer engine never fetches a site, no amount of new writing changes that. This dataset is the thermometer for our own visibility work: it is the only measurement we trust about whether the sites are being read at all.
A user agent string can be copied by anyone. Read every figure here as "what our logs say", not as proof of who sent the request.
Data & methods
- Source. Our own nginx combined access logs on the infogap.store server, rotated files included, counted by our own script. No third-party analytics are involved, and the dataset is published with the same definition.
- Definition. One hit is one request line in our nginx access log whose user agent matches a known AI answer-engine crawler. Hits are counted whether or not the URL existed.
- Last updated. 2026-09-26
- Licence. CC BY-NC 4.0 — attribution: InfoGap (infogap.store), link to this page.
Where to go next
- Data pages: https://infogap.store/data/ai_crawlers.json
- Method pages: https://infogap.store/method/how-to-check-our-numbers/
- Hub: https://infogap.store/data/
- Machine-readable: registry.json