What is this?
Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.
Scrapling is an adaptive web scraping framework for Python that handles everything from a single HTTP request to a full-scale concurrent crawl, with built-in anti-bot bypass.Is this a library or a full crawling framework?Both: it is a Python library you import (Fetcher, StealthyFetcher, DynamicFetcher, parser) and also includes a Scrapy-like Spider framework for full concurrent crawls, plus a CLI and interactive shell.
Both: it is a Python library you import (Fetcher, StealthyFetcher, DynamicFetcher, parser) and also includes a Scrapy-like Spider framework for full concurrent crawls, plus a CLI and interactive shell.How do I get it running?Install it as a Python package and use the Fetcher, StealthyFetcher, or DynamicFetcher classes to make requests, or use the built-in CLI to scrape a URL without writing code.
Install it as a Python package and use the Fetcher, StealthyFetcher, or DynamicFetcher classes to make requests, or use the built-in CLI to scrape a URL without writing code.Do I need Playwright or Chrome installed for the stealth features?For dynamic/stealth fetching it uses Playwright's Chromium or Google Chrome under the hood; plain HTTP fetching doesn't need a browser.
For dynamic/stealth fetching it uses Playwright's Chromium or Google Chrome under the hood; plain HTTP fetching doesn't need a browser.Does it always bypass Cloudflare and other anti-bot systems?It specifically targets bypassing Cloudflare Turnstile/Interstitial out of the box, but enterprise-grade protections like Akamai, DataDome, Kasada, and Incapsula are only handled via a third-party sponsor API, not natively.
It specifically targets bypassing Cloudflare Turnstile/Interstitial out of the box, but enterprise-grade protections like Akamai, DataDome, Kasada, and Incapsula are only handled via a third-party sponsor API, not natively.Does it help avoid DNS leaks when scraping through proxies?Yes, it offers optional DNS-over-HTTPS routing through Cloudflare's DoH to prevent DNS leaks when using proxies.
Yes, it offers optional DNS-over-HTTPS routing through Cloudflare's DoH to prevent DNS leaks when using proxies.Can AI agents like Claude or Cursor use this?Yes, Scrapling ships an MCP server exposing HTTP, browser, and stealth-fetch tools to AI chatbots and agents, stripping prompt-injection content from pages before the agent sees them.
Yes, Scrapling ships an MCP server exposing HTTP, browser, and stealth-fetch tools to AI chatbots and agents, stripping prompt-injection content from pages before the agent sees them.What happens if a website changes its layout?Scrapling's parser uses similarity algorithms to relocate elements automatically after site changes, so your selectors keep working without constant rewrites.
Scrapling's parser uses similarity algorithms to relocate elements automatically after site changes, so your selectors keep working without constant rewrites.Can it pause and resume long crawls?Yes, the Spider framework supports checkpoint-based pause/resume: press Ctrl+C for a graceful stop and restart with the same crawl directory to continue where it left off.
Yes, the Spider framework supports checkpoint-based pause/resume: press Ctrl+C for a graceful stop and restart with the same crawl directory to continue where it left off.Can it turn pages into Markdown for LLMs?Yes, page.markdown() converts any page into clean, sanitized Markdown, and the SiteToMarkdownSpider template can crawl an entire site into a Markdown corpus, all without an LLM in the loop.
Yes, page.markdown() converts any page into clean, sanitized Markdown, and the SiteToMarkdownSpider template can crawl an entire site into a Markdown corpus, all without an LLM in the loop.What are similar projects?Scrapegraph-ai, scrapy, maxun, firecrawl, botasaurus, crawl4ai, SeleniumBase, camoufox
Scrapegraph-ai, scrapy, maxun, firecrawl, botasaurus, crawl4ai, SeleniumBase, camoufoxWhich topics is it in?crawling, http3, stealth, webscraping, web-scraper, data-extraction, data, scraping, crawler, async, playwright, web-scraping
crawling, http3, stealth, webscraping, web-scraper, data-extraction, data, scraping, crawler, async, playwright, web-scraping