Say 👋 to UseScraper Crawler

Today we've launched our first UseScraper Product & API: Crawler

With the UseScraper Crawler you can easily scrape all the pages of any website, and save the content of every page to text, markdown or html. You can export the content as JSON or even a single Markdown file, which is ideal for uploading to a custom OpenAI GPT's knowledge.

Crawling websites should be simple, but it's typically not. To crawl thousands or millions of pages you need a scalable architecture. We've built UseScraper from the ground up to handle even the largest crawling jobs. Websites can also be complex, especially new JavaScript heavy sites. By default we scrape with a headless Chrome browser which can render JavaScript on pages before saving the page content.

Our pricing is fair and affordable. There's no monthly fee to get started, and you only pay for what you use. Our price per 1,000 pages is less than half the typical price other's charge today. You can also launch as many simultaneous crawling jobs as you need, and our infrastructure will scale with your needs.

Sign Up for a free UseScraper account and get $25 free credits

Crawl any website today

You can start a crawling job simply from the UseScraper dashboard, or via the /crawler/jobs API endpoint. Enter the URL of the website you want to scrape, feel free to input https://usescraper.com as a test. Click Start Job.

You can then watch the progress of the crawling job live. The results will be shown when it's complete.

Click View next to any page to see the content scraped. You can access all the results via the /jobs/{id}/data API endpoint, or click the "Download markdown file of results" button to get all the content as a single markdown file.

We've even written a guide on How to upload a website's content into an OpenAI GPT's knowledge.

What's next?

Crawler has a host of advanced features which let you control how a website is crawled. You can learn more about all the features by heading to the docs.

This is just the first product of many we have planned for UseScraper. Keep an eye out for the launch of our single page Scraper coming soon.