colly README

The project's README from github.com/gocolly/colly. As one markdown file: README.md.

Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

GoDoc Backers on Open Collective Sponsors on Open Collective build status report card view examples Code Coverage FOSSA Status Twitter URL

Sponsors

NodeMaven: The most efficient proxy provider for Web Scrapping and Automation with the Highest Quality IP on the market.

Why NodeMaven?

Special codes for Colly users:

NodeMaven

SerpApi: Web Search API for your AI apps

Access real-time data from search engines such as Google, Bing, YouTube, Amazon, and more in structured JSON or Markdown results via a simple API

SerpApi

Features

Example


import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}

See examples folder for more detailed examples.

Installation

go get github.com/gocolly/colly/v2

Bugs

Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

If you are using Colly in a project please send a pull request to add it to the list.

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

FOSSA Status

colly (gocolly/colly)

encoding
Searching Database…

25.6K ·GitHub
Colly
What is this?Colly is a scraping and crawling framework for Go: a clean interface for writing any kind of crawler, scraper or spider that extracts structured data from websites, for things like data mining, processing or archiving.
Is Colly an app I run, or a library I code against?It is a Go framework — you import it into your own Go program and write the crawler in code. There is no GUI or standalone binary to run.
How do I add it to a project?Run 'go get github.com/gocolly/colly/v2' inside your Go project, then look at the examples folder in the repo for ready-made crawler patterns to start from.
Do I need to know Go before using it?Yes — Colly is a Go framework, so your scraper is a Go program you write yourself; there is no GUI or separate scripting layer on top.
What have people actually built with it?A Wappalyzer-style tech detector (gowap), a Steam games database, a website cloner (goclone), an email-scraping CLI, and docker-slim uses it to optimize containers.
Where do I find example scrapers to learn from?The examples folder in the Colly repository contains more detailed examples beyond the basic snippet in the README — a good place to copy your first crawler from.
Why does the install path end in /v2?Go modules require a major-version suffix once a library reaches v2, so you install and import github.com/gocolly/colly/v2 to get the current major version.
What are similar projects?crawlee, scrapy, webmagic, MediaCrawler, Scrapling, proxy_pool, avbook, crawlab
Which topics is it in?crawling, spider, encoding, data-mining, distributed, scraper, scraping, crawler, configuration, caching, concurrency, async
Creator
gocolly
Size
9.5 MB
Stars
25.6K
Updated
CommitOct 16Releasev2.2.0Mar 27
Age
9y
Commits · 90 Days