What is this?
An open-source, agent-oriented web data extraction platform — a self-deployable Firecrawl alternative that turns public pages into clean Markdown, a links tree, and LLM-friendly metadata.
An open-source, agent-oriented web data extraction platform — a self-deployable Firecrawl alternative that turns public pages into clean Markdown, a links tree, and LLM-friendly metadata.Is this a hosted API I sign up for, like Firecrawl, or something I run myself?You run it yourself: the whole platform — a Next.js dashboard, API workers, auth workers, and a database — is open source, and you deploy it to Cloudflare Workers or Vercel.
You run it yourself: the whole platform — a Next.js dashboard, API workers, auth workers, and a database — is open source, and you deploy it to Cloudflare Workers or Vercel.How do I get my own instance running?Clone the repo (a pnpm + Turborepo monorepo), run 'pnpm install' with Node.js 20+, and use 'turbo dev' locally; the repo also ships GitHub Actions workflows that deploy the auth and API workers.
Clone the repo (a pnpm + Turborepo monorepo), run 'pnpm install' with Node.js 20+, and use 'turbo dev' locally; the repo also ships GitHub Actions workflows that deploy the auth and API workers.Is it safe to use in production today?Not yet — the README warns against production use: Deepcrawl is under rapid development, subject to change, and used at your own risk.
Not yet — the README warns against production use: Deepcrawl is under rapid development, subject to change, and used at your own risk.Do I need to be building AI agents for this to be useful?Mainly developers building AI agents — it is optimized for high-frequency agent workloads that scrape public pages and need clean Markdown plus a links tree as model context.
Mainly developers building AI agents — it is optimized for high-frequency agent workloads that scrape public pages and need clean Markdown plus a links tree as model context.Is it really 100% free, or is there a catch?Yes, it is free — MIT-licensed per the repo metadata, and the README stresses open source, open code; the only real cost is your own Cloudflare or Vercel hosting.
Yes, it is free — MIT-licensed per the repo metadata, and the README stresses open source, open code; the only real cost is your own Cloudflare or Vercel hosting.What does Deepcrawl return for each page it crawls?Cleaned Markdown of the page content, an agent-favoured hierarchical links tree, and metadata — packaged so LLMs digest it with minimal token cost, which reduces context switching and hallucination.
Cleaned Markdown of the page content, an agent-favoured hierarchical links tree, and metadata — packaged so LLMs digest it with minimal token cost, which reduces context switching and hallucination.Why pick Deepcrawl over Firecrawl?The author pitches it as a 100% free, fully open-source alternative with better performance, flexibility, and links extraction for agents — and since the entire platform (dashboard, workers, database) is open, you can run your own instance on Cloudflare or Vercel.
The author pitches it as a 100% free, fully open-source alternative with better performance, flexibility, and links extraction for agents — and since the entire platform (dashboard, workers, database) is open, you can run your own instance on Cloudflare or Vercel.Can it get past anti-bot walls or CAPTCHAs?No — the README explicitly says Deepcrawl does not target anti-scraping or anti-bot purposes; it is built for high-frequency scraping of public pages.
No — the README explicitly says Deepcrawl does not target anti-scraping or anti-bot purposes; it is built for high-frequency scraping of public pages.What is the stack under the hood?TypeScript end to end in a pnpm + Turborepo monorepo: a Next.js dashboard, Hono-based API and auth workers, oRPC for the API layer, and Better Auth for authentication, all requiring Node.js 20+.
TypeScript end to end in a pnpm + Turborepo monorepo: a Next.js dashboard, Hono-based API and auth workers, oRPC for the API layer, and Better Auth for authentication, all requiring Node.js 20+.What are similar projects?maxun, firecrawl, Scrapegraph-ai, hono, Scrapling, react-starter-kit, scrapy, all-set-tw
maxun, firecrawl, Scrapegraph-ai, hono, Scrapling, react-starter-kit, scrapy, all-set-twWhich topics is it in?ai-sdk, crawling, hono, web-scraper, ai-agent-tools, data-extraction, crawler, vercel, cloudflare-workers, serverless, cloudflare, web-scraping
ai-sdk, crawling, hono, web-scraper, ai-agent-tools, data-extraction, crawler, vercel, cloudflare-workers, serverless, cloudflare, web-scraping