11.4K ·GitHub
What is this?
Spider-flow is a next-generation crawler platform that lets you design web scraping workflows as flowcharts instead of writing code.
Spider-flow is a next-generation crawler platform that lets you design web scraping workflows as flowcharts instead of writing code.Is this a no-code tool or do I still need to write scraping code?It's a no-code, drag-and-drop platform: you build crawlers visually in a flowchart designer, though it also supports plugin extensions with custom executors and functions for advanced cases.
It's a no-code, drag-and-drop platform: you build crawlers visually in a flowchart designer, though it also supports plugin extensions with custom executors and functions for advanced cases.How do I get it running?The project ships a Docker image, or you can run the built Java application directly since it's a standard Java server app.
The project ships a Docker image, or you can run the built Java application directly since it's a standard Java server app.Do I need a database to save scraped data?Spider-flow can auto-save results to a database or to files, and supports multiple data sources with SQL select/insert/update/delete operations, but a database is optional rather than mandatory.
Spider-flow can auto-save results to a database or to files, and supports multiple data sources with SQL select/insert/update/delete operations, but a database is optional rather than mandatory.Is there anything I should know before scraping with this?The project explicitly warns not to use it for anything that violates laws or ethics, and puts legal responsibility for misuse entirely on the user.
The project explicitly warns not to use it for anything that violates laws or ethics, and puts legal responsibility for misuse entirely on the user.What selectors does it support for extracting data?It supports Xpath, JsonPath, CSS selectors, regex extraction, and mixed combinations of these methods.
It supports Xpath, JsonPath, CSS selectors, regex extraction, and mixed combinations of these methods.Can it handle JavaScript-rendered or AJAX pages?Yes, it can crawl pages rendered by JavaScript or loaded via AJAX, typically using the Selenium plugin for browser-based rendering.
Yes, it can crawl pages rendered by JavaScript or loaded via AJAX, typically using the Selenium plugin for browser-based rendering.What plugins are available out of the box?Built-in plugins include Selenium, Redis, OSS, MongoDB, an IP proxy pool, OCR recognition, and email sending.
Built-in plugins include Selenium, Redis, OSS, MongoDB, an IP proxy pool, OCR recognition, and email sending.Is there a separate frontend project?Yes, spider-flow-vue is the companion Vue.js frontend for spider-flow, and the author also maintains the related magic-api project.
Yes, spider-flow-vue is the companion Vue.js frontend for spider-flow, and the author also maintains the related magic-api project.What are similar projects?crawlab, maxun, Scrapling, proxy_pool, crawl4ai, illa-builder, EasySpider, JeecgBoot
crawlab, maxun, Scrapling, proxy_pool, crawl4ai, illa-builder, EasySpider, JeecgBootWhich topics is it in?visual-programming, spider, web-crawler, data-extraction, crawler, low-code, no-code, mongodb, email, ocr, workflow, redis
visual-programming, spider, web-crawler, data-extraction, crawler, low-code, no-code, mongodb, email, ocr, workflow, redis