html-get

html-get returns reliable HTML for a URL, choosing fetch or prerender depending on page needs.

Quick Start

Install:

npm install html-get browserless puppeteer

Minimal usage:

const createBrowserless = require('browserless')
const getHTML = require('html-get')

const browser = createBrowserless()
const context = browser.createContext()

const result = await getHTML('https://example.com', {
  getBrowserless: () => context
})

console.log(result.html)

await context((browserless) => browserless.destroyContext())
await browser.close()

Recommended Workflow

1. Start with default prerender: 'auto'.
2. Set prerender: false for static pages when speed is priority.
3. Enable rewriteUrls: true when downstream parsing needs absolute links.
4. Enable rewriteHtml: true when source pages have broken meta tags.
5. Reuse one browser process and create/destroy contexts per request.

CLI

One-off usage:

npx -y html-get https://example.com

Debug output with mode, timing, and headers:

npx -y html-get https://example.com --debug

Core Options

• getBrowserless (function): required unless prerender: false.
• prerender ('auto' | true | false): mode selector.
• rewriteUrls (boolean): rewrite relative HTML/CSS URLs to absolute.
• rewriteHtml (boolean): normalize common meta-tag mistakes.
• headers (object): request headers for fetch/prerender.
• gotOpts (object): extra options for got in fetch mode.
• puppeteerOpts (object): options passed to browserless evaluate flow.
• serializeHtml (function): custom output serializer from Cheerio instance.
• encoding (string): output encoding, default utf-8.

Output Shape

getHTML(url, opts) resolves to:

• html: serialized HTML (or custom serializer output fields).
• url: final URL.
• statusCode: HTTP status.
• headers: response headers.
• redirects: redirect chain.
• stats: { mode, timing }.

Common Patterns

Force fast fetch mode for known static targets:

const result = await getHTML(url, {
  prerender: false,
  rewriteUrls: true
})

Prepare HTML for metadata extraction:

const page = await getHTML(url, {
  getBrowserless,
  rewriteUrls: true,
  rewriteHtml: true
})

const metadata = await metascraper({ url: page.url, html: page.html })

Custom serializer (avoid returning full HTML):

const result = await getHTML(url, {
  getBrowserless,
  serializeHtml: ($) => ({
    html: $.html(),
    title: $('title').first().text()
  })
})

Reliability Notes

• If getBrowserless is missing and prerender is not false, html-get throws.
• PDF URLs are fetched and can be converted via mutool when available.
• Media URLs are normalized to HTML wrappers (img, video, audio) for consistent downstream parsing.
• For large batch jobs, control concurrency outside html-get and always clean up browser contexts.