What is this?
pyspider is a powerful web spider (crawler) system written in Python: you write crawling scripts in Python and manage your scraping projects through a built-in web interface.
pyspider is a powerful web spider (crawler) system written in Python: you write crawling scripts in Python and manage your scraping projects through a built-in web interface.Is it a library I import into my code, or something I run?Something you run — it is a full crawler system with its own WebUI, not a library. Your crawl logic lives in Python scripts that the system executes, schedules and monitors for you.
Something you run — it is a full crawler system with its own WebUI, not a library. Your crawl logic lives in Python scripts that the system executes, schedules and monitors for you.How do I get it running?Install it with 'pip install pyspider', run the 'pyspider' command, and open the WebUI in your browser. A Dockerfile is also available in the repo.
Install it with 'pip install pyspider', run the 'pyspider' command, and open the WebUI in your browser. A Dockerfile is also available in the repo.Do I need to set up databases or message queues before using it?Not for a quick start — but at scale it plugs into MySQL, MongoDB, Redis, SQLite, Elasticsearch or PostgreSQL (via SQLAlchemy) for storage, and RabbitMQ, Redis or Kombu as its message queue.
Not for a quick start — but at scale it plugs into MySQL, MongoDB, Redis, SQLite, Elasticsearch or PostgreSQL (via SQLAlchemy) for storage, and RabbitMQ, Redis or Kombu as its message queue.Any gotchas before I expose the dashboard?Yes — the WebUI is open to the public by default and can be used to execute arbitrary commands that may harm your system, so keep it on an internal network or enable need-auth for the WebUI.
Yes — the WebUI is open to the public by default and can be used to execute arbitrary commands that may harm your system, so keep it on an internal network or enable need-auth for the WebUI.What can I do from the WebUI?Write and edit crawler scripts, monitor running tasks, manage projects and view scraped results — the whole crawling workflow happens in the browser.
Write and edit crawler scripts, monitor running tasks, manage projects and view scraped results — the whole crawling workflow happens in the browser.Can it crawl pages that rely on JavaScript?Yes — crawling JavaScript pages is listed among its core features, so it is not limited to static HTML.
Yes — crawling JavaScript pages is listed among its core features, so it is not limited to static HTML.How does it handle retries and repeat visits?Tasks support priorities, retries, periodic runs and re-crawling by page age, so you can keep results fresh without rewriting your scripts.
Tasks support priorities, retries, periodic runs and re-crawling by page age, so you can keep results fresh without rewriting your scripts.Does it scale beyond a single machine?Yes — it has a distributed architecture, so crawlers can run across multiple machines; pair it with a message queue and a scalable database backend for larger workloads.
Yes — it has a distributed architecture, so crawlers can run across multiple machines; pair it with a message queue and a scalable database backend for larger workloads.What are similar projects?crawl4ai, MediaCrawler, scrapy, crawlab, maxun, firecrawl, crawlee, botasaurus
crawl4ai, MediaCrawler, scrapy, crawlab, maxun, firecrawl, crawlee, botasaurusWhich topics is it in?web-crawling, data-mining, data-extraction, scraper, scheduler, crawler, webui, web-scraping, distributed-systems, web-app, automation, tool
web-crawling, data-mining, data-extraction, scraper, scheduler, crawler, webui, web-scraping, distributed-systems, web-app, automation, tool