FlareSolverr — Bypass Cloudflare & Anti-Bot Protection

Fetch content from sites protected by Cloudflare, Akamai, or similar anti-bot systems. Runs a real browser in Docker that solves challenges, then returns page HTML via a local API.

Prerequisites

• Docker available in current shell

Setup

# Ensure container is running (idempotent)
~/.claude/skills/flaresolverr/scripts/flaresolverr-ensure.sh

The script pulls and starts the container if needed, or confirms it's already running.

Usage

Fetch a protected URL

~/.claude/skills/flaresolverr/scripts/flaresolverr-fetch.sh "https://example.com/protected-page"

Returns the full page HTML to stdout. Use with pipes or redirect to file:

# Save to file
~/.claude/skills/flaresolverr/scripts/flaresolverr-fetch.sh "https://example.com" > /tmp/page.html

# Multiple URLs
for url in "$URL1" "$URL2" "$URL3"; do
  ~/.claude/skills/flaresolverr/scripts/flaresolverr-fetch.sh "$url" > "/tmp/$(echo "$url" | md5sum | cut -c1-8).html"
done

Python (direct API)

import requests

def fetch_protected(url, timeout=30000):
    r = requests.post("http://localhost:8191/v1", json={
        "cmd": "request.get",
        "url": url,
        "maxTimeout": timeout
    })
    data = r.json()
    if data["status"] == "ok":
        return data["solution"]["response"]
    raise Exception(f"FlareSolverr error: {data}")

Check status

curl -s http://localhost:8191/ | python3 -m json.tool

Parsing Tips

• JSON-LD: Many sites embed structured data in <script type="application/ld+json"> — check for this first
• Main content: Strip <script> and <style> tags, then extract text from the body
• Indeed specifically: Job description is in <div id="jobDescriptionText">, structured data (salary, location, company) is in JSON-LD

Management

docker stop flaresolverr     # Stop
docker start flaresolverr    # Restart
docker rm -f flaresolverr    # Remove (re-run ensure script to recreate)
docker logs flaresolverr --tail 20  # Debug

Limitations

• Sequential requests (~5–10s per URL) — single browser instance
• ~300–500MB RAM
• First request after startup is slower (~10s)
• Some sites may still block beyond Cloudflare