crawl4ai

Malicious tool definition detected

Tool: SKILL.md [1/2] Description: --- name: crawl4ai description: This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines.

Tool: SKILL.md [2/2]

Malicious tool definition detected

Tool: references/cli-guide.md [1/2] Description: # Crawl4AI CLI Guide <!-- Reference: Tier 2 - Command-line interface for Crawl4AI --> ## Table of Contents <!-- Lines 1-24 --> - [Crawl4AI CLI Guide](#crawl4ai-cli-guide) - [Table of Contents](#table-of-contents) - [Installation](#installation) - [Basic Usage](#basic-usage) - [Quick Example - Advanced Usage](#quick-example---advanced-usage) - [Configuration](#configuration) - [Browser Configuration](#browser-configuration) - [Crawler Configuration

Tool: references/cli-guide.md [2/2]

Malicious tool definition detected

Tool: references/complete-sdk-reference.md [1/24] Description: # Crawl4AI Complete SDK Documentation **Generated:** 2025-10-19 12:56 **Format:** Ultra-Dense Reference (Optimized for AI Assistants) **Crawl4AI Version:** 0.7.4 --- ## Navigation - [Installation & Setup](#installation--setup) (lines 22-126) - [Quick Start](#quick-start) (lines 126-517) - [Core API](#core-api) (lines 517-1056) - [Configuration](#configuration) (lines 1612-2330) - [Crawling Patterns](#crawling-patterns) (lines 2330-35

Tool: references/complete-sdk-reference.md [2/24] Description: crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig from crawl4ai import LLMExtractionStrategy class OpenAIModelFee(BaseModel): model_name: str = Field(..., description="Name of the OpenAI model.") input_fee: str = Field(..., description="Fee for input token for the OpenAI model.") output_fee: str = Field( ..., description="Fee for output token for the OpenAI model." ) async def extract_structured_data_using_llm( provider: s

Tool: references/complete-sdk-reference.md [3/24] Description: you have a **long-running** application or need full control of the crawler’s lifecycle. --- ## 3. Primary Method: `arun()` ```python async def arun( url: str, config: Optional[CrawlerRunConfig] = None, ## Legacy parameters for backward compatibility...

Tool: references/complete-sdk-reference.md [4/24] Description: `"js:() => boolean"` e.g. `js:() => document.querySelectorAll('.item').length > 10`. - `mean_delay` & `max_range`: define random delays for `arun_many()` calls.

Tool: references/complete-sdk-reference.md [5/24] Description: matchers**: `lambda url: 'api' in url` - **Mixed patterns**: Combine strings and functions with `MatchMode.OR` or `MatchMode.AND` - **First match wins**: Configs are evaluated in order - `dispatch_result` in each `CrawlResult` (if using concurrency) can hold memory and timing info. - **Important**: Always include a default config (without `url_matcher`) as the last item if you want to handle all URLs.

Tool: references/complete-sdk-reference.md [6/24] Description: the time of completion. - **`peak_memory`** (float): The peak memory usage (in MB) recorded during the task's execution. - **`start_time`** / **`end_time`** (datetime): Time range for this crawling task. - **`error_message`** (str): Any dispatcher- or concurrency-related error encountered.

Tool: references/complete-sdk-reference.md [7/24] Description: - Typically also set `user_data_dir` to point to a folder. 7. **`cookies`** & **`headers`**: - E.g. `cookies=[{"name": "session", "value": "abc123", "domain": "example.com"}]`.

Tool: references/complete-sdk-reference.md [8/24] Description: # 4) Execute the crawl result = await crawler.arun(url="https://example.com/news", config=run_conf) if result.success: print("Extracted content:", result.extracted_content) else: print("Error:", result.error_message) if __name__ == "__main__": asyncio.run(main()) ``` ## 5. Next Steps - [BrowserConfig, CrawlerRunConfig & LLMConfig Reference](../api/parameters.md) - **Custom Hooks & Auth** (Inject JavaScript or handle login forms). - *

Tool: references/complete-sdk-reference.md [9/24] Description: Slows down if you only want text. | | **`delay_before_return_html`** | `float` (0.1) | Additional pause (seconds) before final HTML is captured. Good for last-second updates.

Tool: references/complete-sdk-reference.md [10/24] Description: url.startswith('https://'), # Must be HTTPS "*.org/*", # Must be .org domain lambda url: 'docs' in url # Must contain 'docs' ], match_mode=MatchMode.AND # ALL conditions must match ) # Combined patterns and functions with AND logic secure_docs = CrawlerRunConfig( url_matcher=["https://*", lambda url: '.doc' in url], match_mode=MatchMode.AND # Must be HTTPS AND contain .doc ) # Default config - matches ALL URLs default_config = Crawl

Tool: references/complete-sdk-reference.md [11/24] Description: Processing <!-- Section: lines 2481-3101 --> ## Markdown Generation Basics 1. How to configure the **Default Markdown Generator** 2. The difference between raw markdown (`result.markdown`) and filtered markdown (`fit_markdown`) > > - You know how to configure `CrawlerRunConfig`.

Tool: references/complete-sdk-reference.md [12/24] Description: # Adjust based on your needs verbose=True ) md_generator = DefaultMarkdownGenerator( content_filter=filter, options={"ignore_links": True} ) config = CrawlerRunConfig( markdown_generator=md_generator, ) async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=config) print(result.markdown.fit_markdown) # Filtered markdown content ``` - **Chunk Processing**: Handles large documents by process

Tool: references/complete-sdk-reference.md [13/24] Description: of output (citations, references, etc.). ## Fit Markdown with Pruning & BM25 ## 1. How “Fit Markdown” Works ### 1.1 The `content_filter` In **`CrawlerRunConfig`**, you can specify a **`content_filter`** to shape how content is pruned or ranked before final markdown generation.

Tool: references/complete-sdk-reference.md [14/24] Description: for link analysis and media collection.

Tool: references/complete-sdk-reference.md [15/24] Description: You can combine `css_selector` and `target_elements` in powerful ways to achieve fine-grained control over your output: ```python import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode async def main(): # Target specific content but preserve page context config = CrawlerRunConfig( # Focus markdown on main content and sidebar target_elements=["#main-content", ".sidebar"], # Global filters applied to entire p

Tool: references/complete-sdk-reference.md [16/24] Description: = 'a[data-testid="pagination-next-button"]'; const button = document.querySelector(selector); if (button) button.click(); """ # Wait until new commits appear wait_for_more = """js:() => { const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4'); if (!window.firstCommit && commits.length>0) { window.firstCommit = commits[0].textContent; return false; } // If top commit changes, we have new commits const topNow = commits[0]?

Tool: references/complete-sdk-reference.md [17/24] Description: second timeout per link query="API documentation guide", # Query for contextual scoring score_threshold=0.3, # Only include links scoring above 0.3 verbose=True # Show detailed progress ), # Enable intrinsic scoring (URL quality, text relevance) score_links=True, # Keep output clean only_text=True, verbose=True ) async with AsyncWebCrawler() as crawler: # Crawl a documentation site (great for testing) result = await crawler.arun("ht

Tool: references/complete-sdk-reference.md [18/24] Description: Optimize performance: link_preview_config = LinkPreviewConfig( max_links=20, # ← Reduce number concurrency=10, # ← Increase parallelism timeout=3, # ← Shorter timeout include_patterns=["*/important/*"] # ← Focus on key areas ) ``` ## 3. Domain Filtering Some websites contain hundreds of third-party or affiliate links.

Tool: references/complete-sdk-reference.md [19/24] Description: `RegexExtractionStrategy` for fast pattern matching 3. **Faster & Cheaper**: No API calls or GPU overhead.

Tool: references/complete-sdk-reference.md [20/24] Description: RegexExtractionStrategy.Currency ) config = CrawlerRunConfig(extraction_strategy=strategy) async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.com", config=config ) if result.success: data = json.loads(result.extracted_content) for item in data[:5]: # Show first 5 matches print(f"{item['label']}: {item['value']}") print(f"Total matches: {len(data)}") asyncio.run(extract_with_regex()) ``` ### Av

Tool: references/complete-sdk-reference.md [21/24] Description: token - Generally provides more accurate schemas - Set via environment variable: `OPENAI_API_KEY` 2. **Ollama (`ollama/llama3.3`)** - Open source alternative - No API token required - Self-hosted option - Good for development and testing ### Benefits of Schema Generation ### Best Practices 1. **Choose Provider Wisely**: - Use OpenAI for production-quality schemas - Use Ollama for development, testing, or when you need a self-hosted

Tool: references/complete-sdk-reference.md [22/24] Description: chunk or call. - **`total_usage`**: sum of all chunk calls.

Tool: references/complete-sdk-reference.md [23/24] Description: > 0 else None, css_selector="li.commit-item", js_only=page > 0, cache_mode=CacheMode.BYPASS ) result = await crawler.arun(config=config) print(f"Page {page + 1}: Found {len(result.extracted_content)} commits") await crawler.crawler_strategy.kill_session(session_id) asyncio.run(integrated_js_and_wait_crawl()) ``` 1. **Authentication Flows**: Login and interact with secured pages. 2.

Tool: references/complete-sdk-reference.md [24/24]

Malicious tool definition detected

Tool: references/sdk-guide.md Description: # Crawl4AI Python SDK Guide <!-- Reference: Tier 2 - Python SDK interface for Crawl4AI --> ## Quick Start <!-- Lines 1-60 --> ### Installation ```bash pip install crawl4ai crawl4ai-setup ``` ### Basic First Crawl ```python import asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") print(result.markdown[:500]) asyncio.run(main()) ``` ### With Configurat

Malicious tool definition detected

Tool: scripts/basic_crawler.py Description: #!/usr/bin/env python3 """ Basic Crawl4AI crawler template Usage: python basic_crawler.py <url> """ import asyncio import sys # Version check MIN_CRAWL4AI_VERSION = "0.7.4" try: from crawl4ai.__version__ import __version__ from packaging import version if version.parse(__version__) < version.parse(MIN_CRAWL4AI_VERSION): print(f"⚠️ Warning: Crawl4AI {MIN_CRAWL4AI_VERSION}+ recommended (you have {__version__})") except ImportError: print(f"ℹ️ Crawl4AI

Malicious tool definition detected

Tool: scripts/batch_crawler.py Description: #!/usr/bin/env python3 """ Crawl4AI batch/multi-URL crawler with concurrent processing Usage: python batch_crawler.py urls.txt [--max-concurrent 5] """ import asyncio import sys import json from pathlib import Path from typing import List, Dict, Any # Version check MIN_CRAWL4AI_VERSION = "0.7.4" try: from crawl4ai.__version__ import __version__ from packaging import version if version.parse(__version__) < version.parse(MIN_CRAWL4AI_VERSION): print(f"⚠️

Malicious tool definition detected

Tool: scripts/extraction_pipeline.py [1/2] Description: #!/usr/bin/env python3 """ Crawl4AI extraction pipeline - Three approaches: 1.

Tool: scripts/extraction_pipeline.py [2/2]

Malicious tool definition detected

Tool: tests/README.md Description: # Crawl4AI Skill Tests This directory contains test scripts that verify the accuracy of all code examples in the SKILL.md file.

Malicious tool definition detected

Tool: tests/run_all_tests.py Description: #!/usr/bin/env python3 """ Run all skill tests """ import subprocess import sys from pathlib import Path def run_test(test_file): """Run a single test file""" print(f" {'='*60}") print(f"Running: {test_file}") print('='*60) result = subprocess.run( [sys.executable, test_file], capture_output=False ) return result.returncode == 0 def main(): test_dir = Path(__file__).parent test_files = [ "test_basic_crawling.py", "test_markdown_generation.py", "test_data

Malicious tool definition detected

Tool: tests/test_advanced_patterns.py Description: #!/usr/bin/env python3 """ Test advanced patterns from SKILL.md """ import asyncio from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig async def test_session_management(): """Test session management""" print("Testing session management...") async with AsyncWebCrawler() as crawler: session_id = "test_session" # First crawl with session config1 = CrawlerRunConfig(session_id=session_id) result1 = await crawler.arun("https://exampl

Malicious tool definition detected

Tool: tests/test_basic_crawling.py Description: #!/usr/bin/env python3 """ Test basic crawling examples from SKILL.md """ import asyncio from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig async def test_basic_crawl(): """Test basic crawling setup""" print("Testing basic crawl setup...") # Test from SKILL.md Section 1 browser_config = BrowserConfig( headless=True, viewport_width=1920, viewport_height=1080, user_agent="custom-agent" ) crawler_config = CrawlerRunConfig( page_time

Malicious tool definition detected

Tool: tests/test_data_extraction.py Description: #!/usr/bin/env python3 """ Test data extraction examples from SKILL.md """ import asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy async def test_manual_schema_extraction(): """Test manual CSS/JSON schema extraction""" print("Testing manual schema extraction...") # Schema from SKILL.md schema = { "name": "articles", "baseSelector": "

Malicious tool definition detected

Tool: tests/test_markdown_generation.py Description: #!/usr/bin/env python3 """ Test markdown generation examples from SKILL.md """ import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator async def test_basic_markdown(): """Test basic markdown extraction""" print("Testing basic markdown extraction...") async with AsyncWebCra