Report AI crawler hits

POST /api/v1/crawler-hits

Reports requests your site received, so Essel can show you which AI assistants and crawlers actually read your content.

This endpoint is the integration boundary. The @essel/ai-crawl npm package is one client of it — not a privileged one — so anything that can make an HTTP request can report hits: a PHP hook, a Go service, an nginx log shipper, a Cloudflare Worker, a cron job.

Detection has to happen server-side. AI crawlers do not execute JavaScript, so no browser snippet can see them. If your platform gives you no server-side request hook at all (standard Shopify and Webflow hosting, for example), publish your blog through Essel’s hosted domains instead — those are tracked automatically with nothing to install.

Authentication

Send an ingest-scoped key, created under API keys, as x-api-key or a bearer token:

x-api-key: cp_ingest_xxxxxxxxxxxxxxxxxxxx

Ingest keys (cp_ingest_…) can only report crawler hits. They cannot read your content, which matters because a tracking key ends up in far more places than a content reader — edge middleware, a plugin’s options row, CI config. A content key (cp_live_…) is rejected here with 403 insufficient_scope.

Never ship this key to a browser. The endpoint is server-to-server and sends no CORS headers.

Request

{
  "events": [
    {
      "url": "https://yoursite.com/blog/pricing-guide",
      "userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
      "ip": "20.171.1.5",
      "ipSource": "cf-connecting-ip",
      "method": "GET",
      "statusCode": 200,
      "occurredAt": "2026-07-25T10:31:00Z",
      "eventId": "1f9c6b0e-4a1e-4a1b-9a26-2b0f6d5a1c33"
    }
  ]
}

Up to 50 events per request. Unknown fields are ignored rather than rejected, so a newer client can post richer events to an older server.

You don’t have to pre-filter

Send everything, or filter first — both are supported. If you send ordinary browser traffic we drop it and say so per event. Filtering locally only saves bandwidth; it cannot improve accuracy, because the registry of ~50 crawlers lives here and is updated here. A crawler that launches next month is recognised for you without you shipping anything.

Response

{
  "accepted": 2,
  "dropped": 1,
  "dryRun": false,
  "results": [
    { "index": 0, "status": "accepted", "crawlerSlug": "gptbot", "category": "training", "verification": "verified" },
    { "index": 1, "status": "accepted", "crawlerSlug": "chatgpt-user", "category": "ai_answers", "verification": "unverified" },
    { "index": 2, "status": "dropped", "reason": "not_a_known_crawler" }
  ]
}

Status is 202 Accepted. Every event reports its own outcome — a batch is never all-or-nothing.

Drop reasons

Verification

We poll the vendors’ published IP-range files daily, so verification adds no latency to your request.

Dry run

Add ?dryRun=true to classify without storing. Same response shape, nothing written. Use it while building an integration:

curl -X POST "https://api.essel.ai/api/v1/crawler-hits?dryRun=true" 
  -H "x-api-key: cp_ingest_xxx" 
  -H "content-type: application/json" 
  -d '{"events":[{"url":"https://yoursite.com/","userAgent":"GPTBot/1.2"}]}'

Retries

Send an eventId per event and reuse it when you retry. Duplicates are rejected for 30 days and reported as duplicate, so a retrying log shipper or queue cannot inflate your numbers. Without an eventId every retry counts again.

Limits

• 50 events per request.
• 600 requests per minute per key. A 429 includes Retry-After; batch and retry rather than dropping the events.
• Identify yourself with X-Essel-Client: my-integration/1.0. It shows up in your tracking-status panel and is how we tell integrations apart when data looks wrong.

Compatibility

/v1 is additive-only: no new required fields, no narrowed values, no changed meanings for a reason string. Everything except url and userAgent is optional and degrades to null, so an integration running in a restricted environment — no client IP, no reliable clock — still works. A breaking change would ship as /v2 with /v1 kept alive, not as a flag day.

Examples

PHP / WordPress — drop into your theme’s functions.php. Non-blocking, so page rendering is untouched:

add_action('shutdown', function () {
  $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
  if (!preg_match('/bot|crawler|spider|gpt|-user/i', $ua)) return;

  wp_remote_post('https://api.essel.ai/api/v1/crawler-hits', [
    'blocking' => false,
    'headers'  => [
      'content-type'   => 'application/json',
      'x-api-key'      => ESSEL_INGEST_KEY,
      'x-essel-client' => 'my-wordpress/1.0',
    ],
    'body' => wp_json_encode(['events' => [[
      'url'        => home_url($_SERVER['REQUEST_URI']),
      'userAgent'  => $ua,
      'ip'         => $_SERVER['REMOTE_ADDR'] ?? null,
      'ipSource'   => 'socket',
      'statusCode' => http_response_code(),
    ]]]),
  ]);
});

Python — from a log shipper, batching yesterday’s access log:

import requests, uuid

events = [{
    "url": f"https://yoursite.com{entry.path}",
    "userAgent": entry.user_agent,
    "ip": entry.ip,
    "ipSource": "socket",
    "statusCode": entry.status,
    "occurredAt": entry.timestamp.isoformat(),
    "eventId": str(uuid.uuid5(uuid.NAMESPACE_URL, entry.raw_line)),  # stable across retries
} for entry in parsed_lines]

for batch in (events[i:i + 50] for i in range(0, len(events), 50)):
    requests.post(
        "https://api.essel.ai/api/v1/crawler-hits",
        headers={"x-api-key": KEY, "x-essel-client": "acme-logshipper/1.0"},
        json={"events": batch},
        timeout=10,
    )

Node / TypeScript — or skip all of this and use the package, which batches, retries nothing, and never throws:

npm install @essel/ai-crawl

import { createBotTracker } from "@essel/ai-crawl";

const tracker = createBotTracker({ apiKey: process.env.ESSEL_INGEST_KEY! });

// Next.js middleware, Hono, Express, SvelteKit and Cloudflare Workers adapters
// live under @essel/ai-crawl/adapters/*.
tracker.trackRequest(request, { waitUntil: ctx.waitUntil });