Convert Any Webpage to Markdown: Tested on 40 Sites

To convert a webpage to Markdown, send its URL to the URL to Markdown API and read data.markdown. We ran 40 real sites through it in October: docs, news front pages, blogs, government sites and JavaScript-heavy product pages. 35 came back with usable Markdown, in a median of 0.9 seconds. The other 5 were refused by the site’s bot protection.

One request

curl -X POST https://api.apiverve.com/v1/urltomarkdown \
  -H "x-api-key: $APIVERVE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.apiverve.com/quickstart"}'

That’s our own quickstart page. The response, with the Markdown cut short:

{
  "status": "ok",
  "error": null,
  "data": {
    "url": "https://docs.apiverve.com/quickstart",
    "title": "Quickstart — APIVerve Documentation",
    "markdown": "Docs/Start/Quickstart\n\n# Quickstart\n\n\nGet a working API call in about four minutes. The same request shape works on every endpoint in the catalog.\n\nCopy for LLMView as MarkdownStar\n\n...",
    "wordCount": 1475,
    "imageCount": 0,
    "linkCount": 19
  }
}

Headings, lists, tables and code blocks come through. So does a little page furniture: “Copy for LLMView as MarkdownStar” is three buttons from the page header run together, and the breadcrumb sits above the title. On a docs page that’s a few words of noise in the first chunk, and it’s typical of what you’ll see. includeImages and includeLinks keep image references and hyperlinks; both are off by default and both need a paid plan.

The first thing the API tries is a plain fetch of the HTML. If that comes back blocked, as a bot challenge, or with too little text to be the real page, it renders the page in headless Chrome and converts what the browser sees. You don’t choose; it’s why the call usually takes under a second but sometimes takes several.

What 40 sites taught us

For each URL we called URL to Markdown, called the Website to Text API for comparison, and fetched the raw HTML ourselves. The list was 10 docs pages, 6 news front pages, 5 blog posts and essays, 2 Wikipedia articles, 6 JavaScript-heavy product sites, 2 government sites and 9 others (GitHub, npm, Hacker News, Stack Overflow, Medium, Substack, the New York Times, Etsy and IKEA).

	URL to Markdown	Website to Text
Returned a result	35 of 40	34 of 40
Under 100 words	1	11
Median time (successes)	0.9 s	4.1 s
90th percentile	4.0 s	8.0 s
Credits per page	10	25

The five failures were AP News, Stack Overflow, Medium, the New York Times and Etsy. All five refuse automated requests, and both APIs reported it the same way: HTTP 503 with "error": "The website returned HTTP 403 and could not be read." Our own plain fetch got a block page from all five too. Etsy took 24 seconds to say no.

The one thin Markdown result was Airbnb’s home page: 27 words, because almost nothing on it is text.

The comparison column changed how we’d build the crawler. An older version of this tutorial fell back to Website to Text whenever Markdown came back thin. On these 40 pages that would rarely help. Website to Text is built for articles: it looks for the main block of a page and returns that. On a blog post that works well (on Dan Luu’s web-bloat essay both APIs returned about 4,000 words). On anything else it returns a fragment: 71 words for the TechCrunch front page against 2,116 from URL to Markdown, 23 for BBC News against 1,314, 16 for apple.com. It came back with less than half of Markdown’s word count on 19 of the 34 pages where both worked, it was slower, and it costs two and a half times as much. So the crawler below doesn’t fall back. A page that comes back thin from URL to Markdown gets skipped.

One page went the other way. npm’s package page for Express returned a block page to our plain fetch (9 words) and 886 words through the API, because the API’s browser fallback got through where a bare HTTP request didn’t.

The crawler

It reads the site’s sitemap to find pages, converts each one, skips thin and refused pages, and writes heading-aware chunks to a JSONL file.

// crawl.mjs: turn a site section into chunked Markdown for an LLM or RAG index.
// Node 18+, no dependencies. Run: APIVERVE_API_KEY=... node crawl.mjs https://docs.example.com/guides/
import { createHash } from 'node:crypto';
import { appendFileSync } from 'node:fs';

const API_KEY = process.env.APIVERVE_API_KEY;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function callApi(api, body, { timeoutMs = 30000, retries = 2 } = {}) {
  for (let attempt = 0; ; attempt++) {
    let res, json;
    try {
      res = await fetch(`https://api.apiverve.com/v1/${api}`, {
        method: 'POST',
        headers: { 'x-api-key': API_KEY, 'Content-Type': 'application/json' },
        body: JSON.stringify(body),
        signal: AbortSignal.timeout(timeoutMs),
      });
      json = await res.json();
    } catch (err) {
      if (attempt >= retries) return { ok: false, error: err.message };
      await sleep(1000 * 2 ** attempt);
      continue;
    }
    if (res.ok && json.status === 'ok') return { ok: true, data: json.data };

    const error = json.error || `HTTP ${res.status}`;
    const outOfCredits = res.headers.get('x-api-remaining-credits') === '0';
    // The site refused us (bot protection, 404). Asking again gets the same answer.
    const siteRefused = /website returned HTTP/i.test(error);
    const retryable = !outOfCredits && !siteRefused && (res.status === 429 || res.status >= 500);
    if (!retryable || attempt >= retries) return { ok: false, error, fatal: outOfCredits };
    await sleep(1000 * 2 ** attempt);
  }
}

const MIN_WORDS = 100;

async function extractPage(url) {
  const md = await callApi('urltomarkdown', { url });
  if (!md.ok) return { url, skipped: md.error, fatal: md.fatal };
  if (md.data.wordCount < MIN_WORDS) return { url, skipped: `only ${md.data.wordCount} words` };
  return { url, title: md.data.title, content: md.data.markdown };
}

// Prefer the sitemap: it lists the pages the owner wants indexed, and reading it costs no credits.
async function discover(seed) {
  const scope = new URL(seed);
  try {
    const xml = await (await fetch(new URL('/sitemap.xml', scope), { signal: AbortSignal.timeout(10000) })).text();
    const urls = [...xml.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)].map((m) => m[1])
      .filter((u) => { const x = new URL(u); return x.hostname === scope.hostname && x.pathname.startsWith(scope.pathname); });
    if (urls.length) return urls;
  } catch { /* no sitemap, fall through */ }

  const res = await callApi('linkscraper', { url: seed, includequery: false });
  if (!res.ok) return [seed];
  const found = new Set([seed]);
  for (const link of res.data.links || []) {
    try {
      const u = new URL(link.href, seed);
      u.hash = '';
      if (u.hostname === scope.hostname && u.pathname.startsWith(scope.pathname)) found.add(u.toString());
    } catch { /* malformed href */ }
  }
  return [...found];
}

const MAX_CHARS = 4000; // about 1,000 tokens of English

function chunkPage(page) {
  const chunks = [];
  for (const section of page.content.split(/\n(?=#{1,3} )/)) {
    const heading = section.match(/^#{1,3} (.+)/)?.[1] || page.title;
    let buf = '';
    // Blank lines first, then a hard cut for blocks that are still too big
    // (pages laid out with HTML tables can arrive as a single block).
    const paras = section.split(/\n\n/).flatMap((p) => p.match(/[\s\S]{1,4000}/g) || []);
    for (const para of paras) {
      if (buf && buf.length + para.length > MAX_CHARS) {
        chunks.push({ url: page.url, title: page.title, heading, text: buf.trim() });
        buf = '';
      }
      buf += para + '\n\n';
    }
    if (buf.trim()) chunks.push({ url: page.url, title: page.title, heading, text: buf.trim() });
  }
  return chunks;
}

async function crawl(seed, { maxPages = 25, delayMs = 1500 } = {}) {
  const urls = (await discover(seed)).slice(0, maxPages);
  const hashes = new Set();
  const fetchedAt = new Date().toISOString();
  console.log(`${urls.length} pages to fetch`);

  for (const url of urls) {
    const page = await extractPage(url);
    if (page.fatal) { console.error(`stopping: ${page.skipped}`); break; }
    if (page.skipped) { console.warn(`skip  ${url} (${page.skipped})`); continue; }

    const hash = createHash('sha256').update(page.content).digest('hex');
    if (hashes.has(hash)) { console.warn(`dupe  ${url}`); continue; }
    hashes.add(hash);

    const chunks = chunkPage(page);
    for (const c of chunks) appendFileSync('chunks.jsonl', JSON.stringify({ ...c, fetchedAt }) + '\n');
    console.log(`ok    ${url} (${chunks.length} chunks)`);
    await sleep(delayMs);
  }
}

await crawl(process.argv[2], { maxPages: Number(process.argv[3] || 25) });

Here it is on the authentication section of our docs, and then on a site that refuses bots:

$ node crawl.mjs https://docs.apiverve.com/authentication 3
3 pages to fetch
ok    https://docs.apiverve.com/authentication (11 chunks)
ok    https://docs.apiverve.com/authentication/key-expiration (11 chunks)
ok    https://docs.apiverve.com/authentication/key-rotation (11 chunks)

$ node crawl.mjs https://medium.com/ 1
1 pages to fetch
skip  https://medium.com/ (The website returned HTTP 403 and could not be read.)

Each line in chunks.jsonl looks like this (text shortened):

{"url":"https://docs.apiverve.com/authentication","title":"Authentication — APIVerve Documentation","heading":"The header","text":"## The header\n\n`x-api-key` is the canonical form, and header names are case-insensitive...","fetchedAt":"2026-10-23T18:26:58.642Z"}

Why it retries less and skips more

The retry rules come straight from the test. A refused site comes back as a 503, and a generic “retry every 5xx” helper (which is what this tutorial used to have) would ask Etsy three times and wait over a minute to be told no each time. So the helper reads the error and gives up at once when the site itself refused. It also stops the whole crawl when your credits run out, which our API signals with a 429 and x-api-remaining-credits: 0. Retrying that is pointless until your renewal date. A 429 without that header is a rate limit, and backing off is the right response.

Discovery reads sitemap.xml before anything else; on most docs sites it’s complete. The Link Scraper API is the fallback for sites without one, and at 25 credits per page it’s the expensive way to find URLs.

Thin pages are skipped rather than retried through another API, for the reasons in the table. MIN_WORDS = 100 is a starting point; a 40-word page is almost always a navigation stub, a login wall or an error page. Tune it on your own corpus.

Chunks split on headings, then on blank lines, then by brute force at 4,000 characters. The brute-force step exists because of one result in the test: Hacker News and Paul Graham’s essays are laid out with HTML tables, and both came back as one enormous Markdown table with no blank lines in it. Without the hard cut, the essay became a single 200,000-character chunk.

Every chunk keeps its url and heading, for showing a source when the model cites one, and fetchedAt, for deciding when to crawl again.

Where it breaks

Sites that block bots stay blocked. 5 of our 40 refused us, and they’re the big publishers and marketplaces you’d expect. Respect that rather than routing around it, and read a site’s robots.txt and terms before you crawl it; they apply whether or not an API does the fetching.

Table-layout pages convert badly. Besides the giant-table problem above, the Paul Graham essay came back with its full text three times in one response, and paragraphs separated by <br> tags ran together. That’s a bug on our side. If a page’s wordCount is far above what you’d expect for it, look at the output before you index it.

Page furniture leaks into the first chunk: breadcrumbs, button labels, the odd video length. On the pages we looked at it was a handful of words per page.

Pages that are mostly pictures or app shells, like Airbnb’s home page, have nothing to extract. The word-count check catches them.