XCrawl Notes

Notes on feeding live web data to LLMs, RAG pipelines and AI agents: scraping, crawling and search.

Feeding live web pages to an LLM without running your own scraper

PENG JIAFU
PENG JIAFU · 2 min read

Most RAG and agent projects hit the same wall early: the model needs fresh content from the web, and the quick scraper you wrote turns into a side project. Pages render with JavaScript, layouts change, and raw HTML wastes tokens. Here is the approach I use for small projects.

1. Fetch pages as Markdown, not HTML

LLMs read Markdown well and it is far shorter than HTML. A web data API such as XCrawl takes a URL and returns clean Markdown (or JSON, HTML, a link list or a screenshot), with JavaScript rendering done on its side. That alone removes most of the cleanup code.

2. Map before you crawl

Before pulling a whole documentation site, list its URLs first. A map call returns the URLs a domain exposes, so you can filter to /docs/ or /blog/ and skip login pages, tag archives and duplicates. Then crawl only that scope with a depth limit.

3. Use search results as an entry point

For agent tasks such as finding the latest pricing page for a product, start from a search API that returns structured results, then scrape the top hits. Structured SERP data is much easier than parsing a results page yourself.

4. Chunk, embed, refresh

Split the Markdown by headings, embed the chunks, and store the source URL with each one. Schedule a re-crawl for pages that change often (pricing, changelogs) and leave stable pages alone.

5. Keep it inside your tools

If you work in Claude Code, Codex or another MCP client, an MCP server lets the assistant call scrape and search directly. XCrawl ships one, plus Python and Node.js SDKs, a CLI and an n8n node, so the same calls work in scripts and no-code flows. New accounts get free credits to test a small knowledge base end to end.

Docs: docs.xcrawl.com ยท Code: github.com/xcrawl-api