FetchLens Docs

Convert your website into AI agentic in a few clicks — deploy a Lens on your verified domain, or install middleware for bot detection.

Create a LensInstall middleware

What is a Lens?

Your website, made agent-operable

A Lens is a verified AI layer on top of your site. FetchLens crawls your public pages, stores them securely, and exposes them through an MCP server (for Cursor, Claude, ChatGPT) and an embeddable widget (for visitors). Answers are grounded in your content — not generic model guesses.

  • Ask & find tools — AI searches and answers from your crawled site
  • Branded widget — chat bubble with your logo, color, and suggested prompts
  • Lens Flows — guided journeys like lead capture, validation, and webhooks
  • Verified ownership — only the domain owner can deploy a Lens for that site

What website owners miss without a Lens

AI agents already visit your site — GPTBot, ClaudeBot, Perplexity, and more. Middleware shows you the traffic. But without a Lens, you leave the conversation to someone else:

  • ×Visitors ask ChatGPT about your product and get wrong or outdated answers — because there is no authoritative endpoint tied to your site.
  • ×Developers cannot plug your site into AI tools — no MCP URL means no structured ask/find over your content.
  • ×High-intent questions (“pricing”, “book a demo”) hit a static FAQ or bounce — no guided flow to capture the lead.
  • ×You track AI traffic in analytics but cannot convert it — agents read your pages; visitors still hunt for answers manually.

A Lens closes that gap: your site speaks for itself, in chat and in AI clients, with proof you own the domain.

Create your first Lens

About 10 minutes from sign-up to live widget

  1. 1

    Sign in and add your domain

    Open the dashboard, add your site, and verify ownership via DNS TXT record or HTML meta tag. You receive a Site Tag (fl_pub_*).

  2. 2

    Create a Lens

    On a verified domain, click Create Lens. FetchLens auto-crawls your sitemap, chunks your pages, and prepares grounded ask/find tools. Status moves from crawling → ready.

  3. 3

    Configure branding & prompts

    Set display name, widget title, logo, brand color, and 3–5 suggested prompts. Optional: enable a Lens Flow (e.g. Lead capture) in the Flows tab.

  4. 4

    Copy MCP URL + widget embed

    From the dashboard: copy the MCP URL for Cursor/Claude, and paste the widget script before </body> on your site.

  5. 5

    Set allowed origins and go live

    Add your production URL(s) to allowed origins, switch visibility from Draft to Public, and save. The widget JWT is origin-bound; MCP is open with rate limits.

Widget embed (from your dashboard)

<script
  src="https://fetchlens-lens-mt.throbbing-thunder-4d33.workers.dev/{siteId}/widget.js"
  defer></script>

Replace {siteId} with your site id (usually your domain). Title, logo, color, and prompts are injected server-side — no extra attributes required.

MCP URL (for AI clients)

https://fetchlens-lens-mt.throbbing-thunder-4d33.workers.dev/{siteId}/mcp

Add as a remote MCP server in Cursor or Claude Desktop. Tools: ask, find, plus flow tools when enabled.

Ready to deploy? Verify your domain first — Lens creation requires proof of ownership.

Open dashboard

Bot detection & analytics

Optional but recommended: install middleware or the script tag to see AI bot crawls and referral traffic in your dashboard. Verify domain ownership first to receive your Site Tag (fl_pub_*).

Next.js Middleware

Server-side bot detection for Vercel, AWS Amplify, Netlify, self-hosted

@aiedx/fetchlens-next

1. Install

npm install @aiedx/fetchlens-next

2. Create middleware.ts in your project root

import { withFetchLens } from '@aiedx/fetchlens-next'

export const middleware = withFetchLens({
  siteId: 'yourdomain.com',
  apiEndpoint: 'https://fetchlens.ai',
  siteTag: 'fl_pub_your_site_tag',
})

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico|api/).*)'],
}

Every request passes through the middleware. FetchLens identifies AI bots by User-Agent and sends detection events to your dashboard without blocking the response.

Cloudflare Pages & Workers

Edge bot detection for Cloudflare-hosted sites

@aiedx/fetchlens-cloudflare

1. Install

npm install @aiedx/fetchlens-cloudflare

2. Cloudflare Pages — create functions/_middleware.ts

import { createPagesMiddleware } from '@aiedx/fetchlens-cloudflare'

export const onRequest = createPagesMiddleware({
  siteId: 'yourdomain.com',
  apiEndpoint: 'https://fetchlens.ai',
  siteTag: 'fl_pub_your_site_tag',
})

3. Cloudflare Workers (alternative)

import { withFetchLens } from '@aiedx/fetchlens-cloudflare'

const handler = withFetchLens({
  siteId: 'yourdomain.com',
  apiEndpoint: 'https://fetchlens.ai',
  siteTag: 'fl_pub_your_site_tag',
})

export default { fetch: handler }

Runs at the edge on every request. Detection events are sent asynchronously via context.waitUntil() so there is zero impact on response time.

Script Tag

Client-side AI referral detection for any website — WordPress, Hugo, Quarto, Shopify, static HTML

Add to your <head>

<script
  src="https://fetchlens.ai/fetchlens.js"
  data-site="yourdomain.com"
  data-key="fl_pub_your_site_tag"
  data-api="https://fetchlens.ai/api/events"
  defer></script>

Note: The script tag detects AI referrals (when users click links from ChatGPT, Claude, Perplexity, etc.). For full bot crawl detection, combine it with the Next.js or Cloudflare middleware above.

Recommended setup

For complete coverage, use middleware + script tag together:

Middleware

Catches AI bot crawls server-side (GPTBot, ClaudeBot, Bytespider, and 60+ more). Bots never run JavaScript, so this is the only way to see them.

Script tag

Catches AI referrals client-side — when a real user clicks a link from ChatGPT, Claude, or Perplexity to your site.

Spider trap prevention (recommended)

Prevent AI crawlers from flooding non-existent URLs on your site

FetchLens does not generate spider traps. They are caused by your site returning HTTP 200 for paths that don't exist. The fastest preventative measure: ship a real robots.txt and sitemap.xml.

Next.js App Router — app/robots.ts

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/', disallow: ['/api/', '/dashboard/'] },
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Next.js App Router — app/sitemap.ts

import type { MetadataRoute } from 'next'

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    { url: 'https://yourdomain.com',       lastModified: new Date(), changeFrequency: 'weekly',  priority: 1 },
    { url: 'https://yourdomain.com/about', lastModified: new Date(), changeFrequency: 'monthly', priority: 0.5 },
  ]
}

Other frameworks

  • Hugo — built-in sitemap template; add a static/robots.txt
  • Quarto — set sitemap: true in _quarto.yml
  • Astro — use @astrojs/sitemap; add a public/robots.txt
  • Jekyll — use jekyll-sitemap gem; add a robots.txt at the project root

Cloudflare users: Enable AI Crawl Control in your Cloudflare dashboard for an additional enforcement layer that blocks known AI scrapers at the edge before they reach your origin.

npm packages

PackageUse case
@aiedx/fetchlens-nextNext.js middleware
@aiedx/fetchlens-cloudflareCloudflare Pages & Workers middleware
@aiedx/fetchlens-coreShared bot signatures & utilities (for custom integrations)
HomeDocsDashboardPrivacyTerms