Bright Data - Scrape

Run the gates below to pick the product yourself - never ask the user which. State the pick in one line, and one word from the user overrides it.

The gates - run in order, first yes wins

Check	If yes
The marketplace check - already collected. All three at once: a huge generic corpus, needed once, and a few months old is fine. Anything less falls through to gate 1.	The `datasets` skill. Buying the download is not scraping, so it has its own skill.
Gate 1 - library. A ready scraper already returns these fields. Scrapers come first. Popular platforms resolve from the bundled top-25 table with zero API calls, and only an unlisted site goes to the live catalogue. The input is not always a URL - variants take usernames, hashtags, keywords, locations and more.	Web Scraper API.
Gate 2 - build it. No ready scraper covers these fields, and any one of these: many pages share one layout, the job will run again, or the data only appears after browser actions (typing, scrolling, waiting, CAPTCHA) and the user has no Playwright, Puppeteer or Selenium setup.	Scraper Studio.
Fall-through. One-time job, no ready scraper, and the pages share no layout.	Back to `fetch` (Web Unlocker), and the user owns the parser.

Many pages means many URLs. One page holding many items is still one page, and with no ready scraper and no recurrence it falls through to fetch, which is a normal outcome, not a failure.

Gate 1 in practice. Open references/web-scraper-api.md and check its bundled top-25 table first, which costs zero API calls. When nothing matches, run node scripts/find-scraper.mjs <query> with a gd_ id, a domain, or part of a scraper name - free reads of the scrapers catalogue (GET /datasets/v3/scrapers, filters ?dataset_id= and ?domain=), junk rows skipped. An id prints every variant's typed inputs, output fields (first 8, all with --json) and trigger line on its own, --sample adds the sample input, a name or domain needs --schema for the same, and --variant <door> narrows it to one door. Manual fallbacks for the same lookup: bdata pipelines search "<site or data type>", or curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY" "https://api.brightdata.com/datasets/v3/scrapers?domain=<site>". All are free reads that trigger no job. An id the user or a colleague handed over gets the same treatment - resolve it with the script before triggering, because a handed id can be a marketplace row, stale, or the wrong variant. With the CLI, any scraper in the catalogue runs by id: bdata pipelines run <dataset_id> <method> "<url>". An id the CLI rejects as unknown runs only through REST or scripts/trigger.mjs. Without the CLI, node scripts/trigger.mjs <dataset_id> "<url>" starts the job and node scripts/poll.mjs <snapshot_id> brings back the records. One URL can return many records (a profile's posts, a product's reviews), each one billed. --limit <n> is sent as limit_per_input, documented for discovery requests, so check the record count of the first run. Run the check and pick the scraper yourself, rather than asking the user which one to use. A single URL is not a reason to skip it. Web Scraper API has no built-in scheduler, so "daily" means the agent writes a cron job or a GitHub Actions workflow in the user's project.

When the user has no URLs. Many ready scrapers take a discovery input instead: a keyword, a category URL, a best-sellers URL, a location. Read the scraper's catalogue entry, its input_schema and discovery doors, before assuming it needs a URL; metadata lists outputs only. Sites with no ready scraper go to Scraper Studio, which covers single-page, search-discovery and multi-page patterns.