Firecrawl Scraping

Overview

Scrape individual web pages and convert them to clean, LLM-ready markdown. Handles JavaScript rendering, anti-bot protection, and dynamic content.

Quick Decision Tree

What are you scraping?
│
├── Single page (article, blog, docs)
│   └── references/single-page.md
│   └── Script: scripts/firecrawl_scrape.py
│
└── Entire website (multiple pages, crawling)
    └── references/website-crawler.md
    └── (Use Apify Website Content Crawler for multi-page)

Environment Setup

# Required in .env
FIRECRAWL_API_KEY=fc-your-api-key-here

Get your API key: https://firecrawl.dev/app/api-keys

Common Usage

Simple Scrape

python scripts/firecrawl_scrape.py "https://example.com/article"

With Options

python scripts/firecrawl_scrape.py "https://wsj.com/article" \
  --proxy stealth \
  --format markdown summary \
  --timeout 60000

Proxy Modes

Mode	Use Case
`basic`	Standard sites, fastest
`stealth`	Anti-bot protection, premium content (WSJ, NYT)
`auto`	Let Firecrawl decide (recommended)

Output Formats

markdown - Clean markdown content (default)
html - Raw HTML
summary - AI-generated summary
screenshot - Page screenshot
links - All links on page

Cost

~1 credit per page. Stealth proxy may use additional credits.

Security Notes

Credential Handling

Store FIRECRAWL_API_KEY in .env file (never commit to git)
API keys can be regenerated at https://firecrawl.dev/app/api-keys
Never log or print API keys in script output
Use environment variables, not hardcoded values

Data Privacy

Only scrapes publicly accessible web pages
Scraped content is processed by Firecrawl servers temporarily
Markdown output stored locally in .tmp/ directory
Screenshots (if requested) are stored locally
No persistent data retention by Firecrawl after request

Access Scopes

API key provides full access to scraping features
No granular permission scopes available
Monitor usage via Firecrawl dashboard

Compliance Considerations

Robots.txt: Firecrawl respects robots.txt by default
Public Content Only: Only scrape publicly accessible pages
Terms of Service: Respect target site ToS
Rate Limiting: Built-in rate limiting prevents abuse
Stealth Proxy: Use stealth mode only when necessary (paywalled news, not auth bypass)
GDPR: Scraped content may contain PII - handle accordingly
Copyright: Respect intellectual property rights of scraped content

Troubleshooting

Common Issues

Issue: Credits exhausted

Symptoms: API returns "insufficient credits" or quota exceeded error Cause: Account credits depleted Solution:

Check credit balance at https://firecrawl.dev/app
Upgrade plan or purchase additional credits
Reduce scraping frequency
Use basic proxy mode to conserve credits

Issue: Page not rendering correctly

Symptoms: Empty content or partial HTML returned Cause: JavaScript-heavy page not fully loading Solution:

Enable JavaScript rendering with --js-render flag
Increase timeout with --timeout 60000 (60 seconds)
Try stealth proxy mode for protected sites
Wait for specific elements with --wait-for selector

Issue: 403 Forbidden error

Symptoms: Script returns 403 status code Cause: Site blocking automated access Solution:

Enable stealth proxy mode
Add delay between requests
Try at different times (some sites rate limit by time)
Check if site requires login (not supported)

Issue: Empty markdown output

Symptoms: Scrape succeeds but markdown is empty or malformed Cause: Dynamic content loaded after page load, or unusual page structure Solution:

Increase wait time for JavaScript to execute
Use --wait-for to wait for specific content
Try html format to see raw content
Check if content is in an iframe (not always supported)

Issue: Timeout errors

Symptoms: Request times out before completion Cause: Slow page load or large page content Solution:

Increase timeout value (up to 120000ms)
Use basic proxy for faster response
Target specific page sections if possible
Check if site is experiencing issues

Resources

references/single-page.md - Single page scraping details
references/website-crawler.md - Multi-page website crawling

Integration Patterns

Scrape and Analyze

Skills: firecrawl-scraping → parallel-research Use case: Scrape competitor pages, then analyze content strategy Flow:

Scrape competitor website pages with Firecrawl
Convert to clean markdown
Use parallel-research to analyze positioning, messaging, features

Scrape and Document

Skills: firecrawl-scraping → content-generation Use case: Create summary documents from web research Flow:

Scrape multiple article pages on a topic
Combine markdown content
Generate summary document via content-generation

Scrape and Enrich CRM

Skills: firecrawl-scraping → attio-crm Use case: Enrich company records with website data Flow:

Scrape company website (about page, team page, product pages)
Extract key information (funding, team size, products)
Update company record in Attio CRM with enriched data