HyperCrawl

You point HyperCrawl at a set of URLs and it crawls them for your LLM or RAG pipeline, tracking which addresses it has visited along the way.

HyperCrawl is an AI web scraping and crawling tool built for LLM and RAG development, and its main selling point is a reduction in retrieval time compared with slower crawling setups.

It comes in two forms.

The Python library is open source and free to use, so you can drop it into an existing project and run it locally.

If your work is web-based, HyperCrawl is also available as an API.

Resource handling is meant to be efficient, which matters when you are crawling many pages for a retrieval index.

The catch is setup.

Expect to need some technical experience to configure it well, and advanced configurations may send you to the documentation.

Results also depend on the sites you crawl, since page structure and crawling restrictions can limit what you get back.

For developers building RAG systems, it is a free option to test.