Firecrawl Deploy Integration

Overview

Deploy applications using Firecrawl's web scraping API (api.firecrawl.dev) to production. Covers API key management, webhook endpoint deployment for async crawl results, and self-hosted Firecrawl deployment options using Docker.

Prerequisites

Firecrawl API key stored in FIRECRAWL_API_KEY environment variable
Application using @mendable/firecrawl-js SDK
Platform CLI installed (vercel, docker, or gcloud)
Webhook endpoint for async crawl results

Instructions

Step 1: Configure Secrets

# Vercel
vercel env add FIRECRAWL_API_KEY production

# Cloud Run
echo -n "your-key" | gcloud secrets create firecrawl-api-key --data-file=-

Step 2: Deploy Scraping API

// api/scrape.ts
import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

export async function POST(req: Request) {
  const { url, formats } = await req.json();

  const result = await firecrawl.scrapeUrl(url, {
    formats: formats || ["markdown"],
  });

  return Response.json({
    markdown: result.markdown,
    metadata: result.metadata,
  });
}

Step 3: Self-Hosted Firecrawl (Docker)

# docker-compose.yml
version: "3.8"
services:
  firecrawl:
    image: mendableai/firecrawl:latest
    ports:
      - "3002:3002"  # 3002 = configured value
    environment:
      - REDIS_URL=redis://redis:6379  # 6379: Redis port
      - PLAYWRIGHT_BROWSERS_PATH=/browsers
    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"  # Redis port

  app:
    build: .
    ports:
      - "3000:3000"  # 3000: 3 seconds in ms
    environment:
      - FIRECRAWL_API_URL=http://firecrawl:3002
    depends_on:
      - firecrawl

Step 4: Webhook Endpoint for Async Crawls

// api/webhooks/firecrawl.ts
export async function POST(req: Request) {
  const { type, id, data } = await req.json();

  if (type === "crawl.completed") {
    await processScrapedPages(id, data.pages);
  }

  return Response.json({ received: true });
}

Step 5: Health Check

export async function GET() {
  try {
    const result = await firecrawl.scrapeUrl("https://example.com", {
      formats: ["markdown"],
    });
    return Response.json({ status: result ? "healthy" : "degraded" });
  } catch {
    return Response.json({ status: "unhealthy" }, { status: 503 });  # HTTP 503 Service Unavailable
  }
}

Error Handling

Issue	Cause	Solution
Rate limited	Too many scrape requests	Queue requests with delays
Scrape blocked	Target site protection	Use `waitFor` and browser options
API key invalid	Key expired	Regenerate at firecrawl.dev dashboard
Self-hosted memory	Playwright overhead	Increase container memory to 2GB+

Examples

Basic usage: Apply firecrawl deploy integration to a standard project setup with default configuration options.

Advanced scenario: Customize firecrawl deploy integration for production environments with multiple constraints and team-specific requirements.

Resources

Next Steps

For webhook handling, see firecrawl-webhooks-events.

Output

Configuration files or code changes applied to the project
Validation report confirming correct implementation
Summary of changes made and their rationale

firecrawl-deploy-integration