brightdata by microck/ordinary-claude-skills

Workflow Routing (SYSTEM PROMPT)

CRITICAL: This workflow implements progressive escalation for URL content retrieval.

When user requests scraping/fetching URL content: Examples: "scrape this URL", "fetch this page", "get content from [URL]", "pull content from this site", "retrieve [URL]", "can't access this site", "this site is blocking me", "use Bright Data to fetch" → READ: ${PAI_DIR}/skills/brightdata/workflows/four-tier-scrape.md → EXECUTE: Four-tier progressive scraping workflow (WebFetch → Curl → Browser Automation → Bright Data MCP)

When to Activate This Skill

Direct Scraping Requests (Categories 1-4)

"scrape this URL", "scrape [URL]", "scrape this page"
"fetch this URL", "fetch [URL]", "fetch this page", "fetch content from"
"pull content from [URL]", "pull this page", "pull from this site"
"get content from [URL]", "retrieve [URL]", "retrieve this page"
"do scraping on [URL]", "run scraper on [URL]"
"basic scrape", "quick scrape", "simple fetch"
"comprehensive scrape", "deep scrape", "full content extraction"

Access & Bot Detection Issues (Categories 5-7)

"can't access this site", "site is blocking me", "getting blocked"
"bot detection", "CAPTCHA", "access denied", "403 error"
"need to bypass bot detection", "get around blocking"
"this URL won't load", "can't fetch this page"
"use Bright Data", "use the scraper", "use advanced scraping"

Result-Oriented Requests (Category 8)

"get me the content from [URL]"
"extract text from [URL]"
"download this page content"
"convert [URL] to markdown"
"need the HTML from this site"

Use Case Indicators

User needs web content for research or analysis
Standard methods (WebFetch) are failing
Site has bot detection or rate limiting
Need reliable content extraction
Converting web pages to structured format (markdown)

Core Capabilities

Progressive Escalation Strategy:

Tier 1: WebFetch - Fast, simple, built-in Claude Code tool
Tier 2: Customized Curl - Chrome-like browser headers to bypass basic bot detection
Tier 3: Browser Automation - Full browser automation using Playwright for JavaScript-heavy sites
Tier 4: Bright Data MCP - Professional scraping service that handles CAPTCHA and advanced bot detection

Key Features:

Automatic fallback between tiers
Preserves content in markdown format
Handles bot detection and CAPTCHA
Works with any URL
Efficient resource usage (only escalates when needed)

Workflow Overview

four-tier-scrape.md - Complete URL content scraping with four-tier fallback strategy

When to use: Any URL content retrieval request
Process: Start with WebFetch → If fails, use curl with Chrome headers → If fails, use Browser Automation → If fails, use Bright Data MCP
Output: URL content in markdown format

Extended Context

Integration Points:

WebFetch Tool - Built-in Claude Code tool for basic URL fetching
Bash Tool - For executing curl commands with custom headers
Browser Automation - Playwright-based browser automation for JavaScript rendering
Bright Data MCP - mcp__Brightdata__scrape_as_markdown for advanced scraping

When Each Tier Is Used:

Tier 1 (WebFetch): Simple sites, public content, no bot detection
Tier 2 (Curl): Sites with basic user-agent checking, simple bot detection
Tier 3 (Browser Automation): Sites requiring JavaScript execution, dynamic content loading
Tier 4 (Bright Data): Sites with CAPTCHA, advanced bot detection, residential proxy requirements

Configuration: No configuration required - all tools are available by default in Claude Code

Examples

Example 1: Simple Public Website

User: "Scrape https://example.com"

Skill Response:

Routes to three-tier-scrape.md
Attempts Tier 1 (WebFetch)
Success → Returns content in markdown
Total time: <5 seconds

Example 2: Site with JavaScript Requirements

User: "Can't access this site https://dynamic-site.com"

Skill Response:

Routes to four-tier-scrape.md
Attempts Tier 1 (WebFetch) → Fails (blocked)
Attempts Tier 2 (Curl with Chrome headers) → Fails (JavaScript required)
Attempts Tier 3 (Browser Automation) → Success
Returns content in markdown
Total time: ~15-20 seconds

Example 3: Site with Advanced Bot Detection

User: "Scrape https://protected-site.com"

Skill Response:

Routes to four-tier-scrape.md
Attempts Tier 1 (WebFetch) → Fails (blocked)
Attempts Tier 2 (Curl) → Fails (advanced detection)
Attempts Tier 3 (Browser Automation) → Fails (CAPTCHA)
Attempts Tier 4 (Bright Data MCP) → Success
Returns content in markdown
Total time: ~30-40 seconds

Example 4: Explicit Bright Data Request

User: "Use Bright Data to fetch https://difficult-site.com"

Skill Response:

Routes to four-tier-scrape.md
User explicitly requested Bright Data
Goes directly to Tier 4 (Bright Data MCP) → Success
Returns content in markdown
Total time: ~5-10 seconds

Related Documentation:

${PAI_DIR}/skills/CORE/SKILL-STRUCTURE-AND-ROUTING.md - Canonical structure guide
${PAI_DIR}/skills/CORE/CONSTITUTION.md - Overall Kai philosophy

Last Updated: 2025-11-23