Agent Analytics: Track AI Crawlers on Your Site | Rankshift
Live tracking for AI crawlers and agents

Agent Analytics:
see which pages AI crawlers read

Rankshift reads your server logs and lists every request from GPTBot, ClaudeBot, PerplexityBot and other AI agents: the page, the status code and the time. Then it sets that next to the pages AI answers cite.

Start your 30-day trial Book a demo
30-day free trial No credit card needed Included on every plan
What you see

The traffic your web analytics leaves out

AI crawlers request your HTML and skip the JavaScript, so GA4 and similar tools do not record them. Your server logs do. Agent Analytics turns those logs into four views.

  • Agents by name

    GPTBot, ClaudeBot, PerplexityBot and others, each listed separately.

  • Pages they request

    Every URL an agent fetched, with a request count per agent.

  • Status codes

    200, 301, 403, 404 or 5xx, for each agent and each page.

  • Activity over time

    Daily requests per agent, so a spike or a drop is easy to spot.

Agents

Every AI agent by name, operator and purpose

GPTBot collects training data, OAI-SearchBot builds the search index behind ChatGPT, and ChatGPT-User fetches a page at the moment someone asks a question. Rankshift reports each one separately, so you can tell training crawls from requests tied to live answers.

  • Grouped by operator, from OpenAI and Anthropic to Perplexity and Google
  • Labeled as training, search or user request
  • Request volume per agent
Pages

Which pages AI agents request most

Sort your URLs by agent requests and see who asked for each one. If ChatGPT-User fetches your pricing page hundreds of times a month, that page is being read during live conversations.

  • Requests per URL, split by agent
  • Change against the previous period
  • The most active agent on every page
Errors

Find the pages AI agents cannot read

A page that answers an AI agent with a 404 or a 403 cannot be quoted. Rankshift lists every URL where an agent hit an error or a redirect, so you know which link, redirect or firewall rule to fix.

  • Status code per agent and per URL
  • Blocked requests (403) kept apart from missing pages (404)
  • Sorted by number of failed requests
How it works

From server log to agent report

  1. Connect your logs

    Step 1

    Link your Cloudflare zone, send logs through the API, or upload an access log file.

  2. We identify the agents

    Step 2

    Each request is matched against known AI agents and labeled with its operator and purpose.

  3. Act on what they read

    Step 3

    See which pages agents request, which ones return errors, and which ones are cited in AI answers.

Setup

Pick the setup that fits your stack

Your server already records every AI agent request. Rankshift only needs a way to read those records.

  • Cloudflare Connect your zone and new requests sync on their own.
    Automatic
  • API Connect directly to our endpoint to submit live data.
    Any source
  • Log file upload Upload an access log and get a first report.
    No setup
FAQ

Questions about Agent Analytics

What is Agent Analytics?

Agent Analytics is the part of Rankshift that reads your server logs and reports the requests made by AI crawlers and agents such as GPTBot, ClaudeBot, ChatGPT-User and PerplexityBot. You see which agents visit, which pages they request and which status code they receive.

How is this different from GA4 or other web analytics?

Web analytics tools rely on JavaScript that runs in a visitor's browser. AI crawlers request the HTML and do not run that script, so they never appear in those reports. Server logs record every request, which is why Agent Analytics works from logs.

What is the difference between training, search and user request agents?

Training crawlers such as GPTBot collect pages that may be used to train future models. Search crawlers such as OAI-SearchBot and PerplexityBot build the index an assistant searches. User request agents such as ChatGPT-User fetch a page at the moment someone asks a question. The last two are tied to the answers people see today.

How do I connect my site?

Connect Cloudflare, submit live data directly to our API endpoint, or upload an access log file. A single upload is enough for a first report.

Does a crawl mean my page will be cited?

No. A crawl means the agent was able to read the page. Whether an answer cites it depends on the prompt and on the other sources available. That is why Rankshift shows crawl data next to your citation data.

Is Agent Analytics included in my plan?

Yes. Agent Analytics is part of the Starter, Professional, Business and Enterprise plans, and you can try it during the 30-day free trial.

See which AI agents visited your site this week

Connect Cloudflare or upload a log file and get your first agent report during the free trial. No credit card needed.

Start your 30-day trial Book a demo