31.9K ·GitHub
What is this?
ScrapeGraphAI is a Python web scraping library that uses LLMs and direct graph logic to build scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown). You just say which information you want to extract and it does the rest.
ScrapeGraphAI is a Python web scraping library that uses LLMs and direct graph logic to build scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown). You just say which information you want to extract and it does the rest.Is it a library I run myself, or a hosted service I call?It is a Python library you run on your own infrastructure, bringing your own LLM keys. There is also a managed cloud API with Python and JS SDKs, but that is the separate paid service, not this repo.
It is a Python library you run on your own infrastructure, bringing your own LLM keys. There is also a managed cloud API with Python and JS SDKs, but that is the separate paid service, not this repo.How do I get it into my project?Install it from PyPI with pip — the README recommends doing that inside a virtual environment to avoid conflicts with other libraries. The repo also ships Docker and docker-compose files.
Install it from PyPI with pip — the README recommends doing that inside a virtual environment to avoid conflicts with other libraries. The repo also ships Docker and docker-compose files.Do I need an OpenAI API key, or can it run on local models?Either: it works with LLM APIs — OpenAI, Groq, Azure, Gemini, MiniMax and more — or fully local models via Ollama. For local models you need Ollama installed and the model pulled with 'ollama pull'.
Either: it works with LLM APIs — OpenAI, Groq, Azure, Gemini, MiniMax and more — or fully local models via Ollama. For local models you need Ollama installed and the model pulled with 'ollama pull'.What is the catch — do I handle browsers and bot-blocking myself?Yes — with the open-source library, JS rendering via Playwright, proxies, anti-bot handling, scaling and maintenance are all your responsibility. The managed API includes those, billed per credit.
Yes — with the open-source library, JS rendering via Playwright, proxies, anti-bot handling, scaling and maintenance are all your responsibility. The managed API includes those, billed per credit.Does it phone home about how I use it?It collects anonymous usage telemetry by default. You can opt out by setting the SCRAPEGRAPHAITELEMETRYENABLED environment variable to false.
It collects anonymous usage telemetry by default. You can opt out by setting the SCRAPEGRAPHAITELEMETRYENABLED environment variable to false.What scraping pipelines does it ship with?Six graph pipelines: SmartScraperGraph (single page from a prompt + source), SearchGraph (top n search results), SpeechGraph (extracts info and returns an audio file), ScriptCreatorGraph (generates a Python script), plus Multi versions of SmartScraper and ScriptCreator for lists of pages with parallel LLM calls.
Six graph pipelines: SmartScraperGraph (single page from a prompt + source), SearchGraph (top n search results), SpeechGraph (extracts info and returns an audio file), ScriptCreatorGraph (generates a Python script), plus Multi versions of SmartScraper and ScriptCreator for lists of pages with parallel LLM calls.Which pipeline should I start with?SmartScraperGraph — it is the most common one and needs only a user prompt and a source URL, returning the extracted data as a dictionary.
SmartScraperGraph — it is the most common one and needs only a user prompt and a source URL, returning the extracted data as a dictionary.When would I pick this over the paid ScrapeGraph cloud API?The library when you want full control, on-prem data, local Ollama models or fine-grained cost tuning, and don't mind managing browsers, proxies and scaling. The cloud API wins for zero infrastructure, managed JS rendering and anti-bot, and the fastest path to production.
The library when you want full control, on-prem data, local Ollama models or fine-grained cost tuning, and don't mind managing browsers, proxies and scaling. The cloud API wins for zero infrastructure, managed JS rendering and anti-bot, and the fastest path to production.What tools and frameworks does it integrate with?LLM frameworks (Langchain, Llama Index, Crew.ai, Agno, CamelAI), low-code platforms (Pipedream, Bubble, Zapier, n8n, Dify, Toolhouse), an MCP server, and official SDKs for Python and Node.
LLM frameworks (Langchain, Llama Index, Crew.ai, Agno, CamelAI), low-code platforms (Pipedream, Bubble, Zapier, n8n, Dify, Toolhouse), an MCP server, and official SDKs for Python and Node.What are similar projects?firecrawl, maxun, Scrapling, crawl4ai, wigolo, MediaCrawler, crawlee, scrapy
firecrawl, maxun, Scrapling, crawl4ai, wigolo, MediaCrawler, crawlee, scrapyWhich topics is it in?large-language-model, ai-search, web-crawler, web-scraper, webscraping, n8n, web-search, data-extraction, scraper, scraping, crawler, python-library
large-language-model, ai-search, web-crawler, web-scraper, webscraping, n8n, web-search, data-extraction, scraper, scraping, crawler, python-library