Install FetchLens

Detect AI bots at the server layer and track AI referral traffic. Choose the integration that matches your stack.

Next.js Middleware

Server-side bot detection for Vercel, AWS Amplify, Netlify, self-hosted

@aiedx/fetchlens-next

1. Install

npm install @aiedx/fetchlens-next

2. Create middleware.ts in your project root

import { withFetchLens } from '@aiedx/fetchlens-next'

export const middleware = withFetchLens({
  siteId: 'yourdomain.com',
  apiEndpoint: 'https://fetchlens.ai',
  siteTag: 'fl_pub_your_site_tag',
})

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico|api/).*)'],
}

Every request passes through the middleware. FetchLens identifies AI bots by User-Agent and sends detection events to your dashboard without blocking the response.

Cloudflare Pages & Workers

Edge bot detection for Cloudflare-hosted sites

@aiedx/fetchlens-cloudflare

1. Install

npm install @aiedx/fetchlens-cloudflare

2. Cloudflare Pages — create functions/_middleware.ts

import { createPagesMiddleware } from '@aiedx/fetchlens-cloudflare'

export const onRequest = createPagesMiddleware({
  siteId: 'yourdomain.com',
  apiEndpoint: 'https://fetchlens.ai',
  siteTag: 'fl_pub_your_site_tag',
})

3. Cloudflare Workers (alternative)

import { withFetchLens } from '@aiedx/fetchlens-cloudflare'

const handler = withFetchLens({
  siteId: 'yourdomain.com',
  apiEndpoint: 'https://fetchlens.ai',
  siteTag: 'fl_pub_your_site_tag',
})

export default { fetch: handler }

Runs at the edge on every request. Detection events are sent asynchronously via context.waitUntil() so there is zero impact on response time.

Script Tag

Client-side AI referral detection for any website — WordPress, Hugo, Quarto, Shopify, static HTML

Add to your <head>

<script
  src="https://fetchlens.ai/fetchlens.js"
  data-site="yourdomain.com"
  data-key="fl_pub_your_site_tag"
  data-api="https://fetchlens.ai/api/events"
  defer></script>

Note: The script tag detects AI referrals (when users click links from ChatGPT, Claude, Perplexity, etc.). For full bot crawl detection, combine it with the Next.js or Cloudflare middleware above.

Recommended setup

For complete coverage, use middleware + script tag together:

Middleware

Catches AI bot crawls server-side (GPTBot, ClaudeBot, Bytespider, and 60+ more). Bots never run JavaScript, so this is the only way to see them.

Script tag

Catches AI referrals client-side — when a real user clicks a link from ChatGPT, Claude, or Perplexity to your site.

Spider trap prevention (recommended)

Prevent AI crawlers from flooding non-existent URLs on your site

FetchLens does not generate spider traps. They are caused by your site returning HTTP 200 for paths that don't exist. The fastest preventative measure: ship a real robots.txt and sitemap.xml.

Next.js App Router — app/robots.ts

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/', disallow: ['/api/', '/dashboard/'] },
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Next.js App Router — app/sitemap.ts

import type { MetadataRoute } from 'next'

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    { url: 'https://yourdomain.com',       lastModified: new Date(), changeFrequency: 'weekly',  priority: 1 },
    { url: 'https://yourdomain.com/about', lastModified: new Date(), changeFrequency: 'monthly', priority: 0.5 },
  ]
}

Other frameworks

  • Hugo — built-in sitemap template; add a static/robots.txt
  • Quarto — set sitemap: true in _quarto.yml
  • Astro — use @astrojs/sitemap; add a public/robots.txt
  • Jekyll — use jekyll-sitemap gem; add a robots.txt at the project root

Cloudflare users: Enable AI Crawl Control in your Cloudflare dashboard for an additional enforcement layer that blocks known AI scrapers at the edge before they reach your origin.

npm packages

PackageUse case
@aiedx/fetchlens-nextNext.js middleware
@aiedx/fetchlens-cloudflareCloudflare Pages & Workers middleware
@aiedx/fetchlens-coreShared bot signatures & utilities (for custom integrations)
HomeDocsDashboardPrivacyTerms