crawlee README

The project's README from github.com/apify/crawlee. As one markdown file: README.md.

Crawlee
A web scraping and browser automation library

apify%2Fcrawlee | Trendshift

NPM latest version Downloads Chat on discord Build Status

Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.

Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.

Crawlee is available as the crawlee NPM package.

👉 View full documentation, guides and examples on the Crawlee project website 👈

Do you prefer 🐍 Python instead of JavaScript? 👉 Checkout Crawlee for Python 👈.

Installation

We recommend visiting the Introduction tutorial in Crawlee documentation for more information.

Crawlee requires Node.js 16 or higher.

With Crawlee CLI

The fastest way to try Crawlee out is to use the Crawlee CLI and choose the Getting started example. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.

npx crawlee create my-crawler

cd my-crawler
npm start

Manual installation

If you prefer adding Crawlee into your own project, try the example below. Because it uses PlaywrightCrawler we also need to install Playwright. It's not bundled with Crawlee to reduce install size.

npm install crawlee playwright

import { PlaywrightCrawler, Dataset } from 'crawlee';

// PlaywrightCrawler crawls the web using a headless
// browser controlled by the Playwright library.
const crawler = new PlaywrightCrawler({
    // Use the requestHandler to process each of the crawled pages.
    async requestHandler({ request, page, enqueueLinks, log }) {
        const title = await page.title();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);

        // Save results as JSON to ./storage/datasets/default
        await Dataset.pushData({ title, url: request.loadedUrl });

        // Extract links from the current page
        // and add them to the crawling queue.
        await enqueueLinks();
    },
    // Uncomment this option to see the browser window.
    // headless: false,
});

// Add first URL to the queue and start the crawl.
await crawler.run(['https://crawlee.dev']);

By default, Crawlee stores data to ./storage in the current working directory. You can override this directory via Crawlee configuration. For details, see Configuration guide, Request storage and Result storage.

Installing pre-release versions

We provide automated beta builds for every merged code change in Crawlee. You can find them in the npm list of releases. If you want to test new features or bug fixes before we release them, feel free to install a beta build like this:

npm install crawlee@next

If you also use the Apify SDK, you need to specify dependency overrides in your package.json file so that you don't end up with multiple versions of Crawlee installed:

{
    "overrides": {
       "apify": {
           "@crawlee/core": "$crawlee",
           "@crawlee/types": "$crawlee",
           "@crawlee/utils": "$crawlee"
       }
    }
}

🛠 Features

👾 HTTP crawling

💻 Real browser crawling

Usage on the Apify platform

Crawlee is open-source and runs anywhere, but since it's developed by Apify, it's easy to set up on the Apify platform and run in the cloud. Visit the Apify SDK website to learn more about deploying Crawlee to the Apify platform.

Support

If you find any bug or issue with Crawlee, please submit an issue on GitHub. For questions, you can ask on Stack Overflow, in GitHub Discussions or you can join our Discord server.

Contributing

Your code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see CONTRIBUTING.md.

License

This project is licensed under the Apache License 2.0 - see the LICENSE.md file for details.

crawlee (apify/crawlee)

Searching Database…

26.2K ·GitHub
Crawlee
What is this?A web scraping and browser automation library for Node.js: you write crawlers in JavaScript or TypeScript to crawl sites, extract data (e.g. for AI, LLM and RAG pipelines) and save it to disk or cloud.
Is this a library I code against, or a ready-made scraping app?A library, not an app: you install the crawlee npm package and write the crawler yourself in JS/TS. The bundled CLI only scaffolds a starter project for you.
How do I set up my first crawler with it?Fastest: run the Crawlee CLI and pick the Getting started example — it installs dependencies and adds boilerplate to play with. Or add the crawlee package to your own Node.js project.
Does it come with the browsers, or do I install those myself?You install Playwright or Puppeteer yourself for browser crawling — Crawlee skips them to keep the install size small. Otherwise you just need Node.js 16 or higher.
Will sites with anti-bot protection block my crawlers?Blocking is exactly what it fights: even with default settings, crawls look human-like and slip past modern bot protections — proxy rotation, session management and browser-like TLS fingerprints are built in.
Does it have to run on the Apify cloud?No — it is open-source and runs anywhere Node.js runs, from your laptop to your own server. The Apify platform is just an optional hosted option from the same team.
Where does the scraped data end up?By default in a ./storage folder inside your working directory; you can override the path in the configuration, and storage is pluggable for both tabular data and files.
Besides HTML pages, what can it grab?JSON APIs are fair game too, and it downloads files such as PDFs and images; in browser mode it renders JavaScript-heavy pages and can take screenshots.
Which browsers and drivers does it support?Playwright and Puppeteer behind a single interface, driving Chrome, Firefox, WebKit and others in headless or headful mode, with automatic browser management.
What are similar projects?moli, crawl4ai, reader, MediaCrawler, scrapy, maxun, chromeless, Scrapegraph-ai
Which topics is it in?web-crawling, crawling, headless-chrome, web-crawler, puppeteer, scraper, scraping, headless, crawler, playwright, web-scraping, browser-automation
Creator
apify
Size
172.7 MB
Stars
26.2K
Updated
CommitOct 15Releasev3.18.2Sep 29
Age
10y
Commits · 90 Days