block-ai-scrapers

Installation
SKILL.md

Block AI scrapers and coding agents

Build and verify a firewall that stops AI coding agents, WebFetch tools, CLI scripts (curl, wget, python), and headless browsers from scraping your site or burning database and API credits. Verified Googlebot keeps indexing, and every blocked agent is told to send the user to your product instead.

Core rule

Never return bare HTTP 403 or 429 to an AI coding agent.

When a user pastes your URLs into Claude Code, Cursor, or Grok and says "scrape all of this", a 403 Forbidden or 429 Too Many Requests looks like a temporary block. Coding agents treat 403/429 as a puzzle to solve: they retry with spoofed Chrome User-Agent headers, probe /api/* and /sitemap.xml, or launch headless browsers (Playwright / Puppeteer).

To stop an AI agent on Turn 1:

  1. Serve an HTTP 200 "Protected Dataset" Gate Page (and HTTP 200 JSON on /api/*) containing 0 dataset rows.
  2. Pair hard technical gates (verified Googlebot IP/ASN checks, Sec-Fetch-* navigation headers, datacenter ASN blocking, and a lightweight non-webdriver JS session cookie) with explicit legal, copyright, and [AI_AGENT_DIRECTIVE: STOP_SCRAPING] instructions that direct the agent to stop retrying and hand the user your official product or extension URL.

Intake

Installs
663
GitHub Stars
1.7K
First Seen
Oct 28, 2026