colly README

The project's README from github.com/gocolly/colly. As one markdown file: README.md.

Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

GoDoc Backers on Open Collective Sponsors on Open Collective build status report card view examples Code Coverage FOSSA Status Twitter URL

Sponsors

NodeMaven: The most efficient proxy provider for Web Scrapping and Automation with the Highest Quality IP on the market.

Why NodeMaven?

Special codes for Colly users:

NodeMaven

SerpApi: Web Search API for your AI apps

Access real-time data from search engines such as Google, Bing, YouTube, Amazon, and more in structured JSON or Markdown results via a simple API

SerpApi

Features

Example


import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}

See examples folder for more detailed examples.

Installation

go get github.com/gocolly/colly/v2

Bugs

Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

If you are using Colly in a project please send a pull request to add it to the list.

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

FOSSA Status

colly (gocolly/colly)

Searching Database…

25.6K ·GitHub
这是干什么的?Colly 是一个 Go 语言的数据抓取和爬虫框架:它提供一套简洁的接口,让你编写各类爬虫、抓取器或蜘蛛程序,从网站中提取结构化数据,可用于数据挖掘、数据处理或归档等场景。
Colly 是一个我能直接运行的应用,还是写代码时调用的库?它是一个 Go 框架——你需要把它导入自己的 Go 程序里,用代码来写爬虫。它没有图形界面,也没有可以直接运行的独立程序。
怎么把它加到我的项目里?在你的 Go 项目中运行 'go get github.com/gocolly/colly/v2',然后查看仓库里的 examples 文件夹,里面有现成的爬虫示例可以参考起步。
用之前我得先会 Go 吗?是的——Colly 是一个 Go 框架,所以你的抓取程序就是一个你自己编写的 Go 程序;它没有图形界面,也没有额外的脚本层。
大家用它做出过什么东西?比如一个类似 Wappalyzer 的网站技术检测工具(gowap)、一个 Steam 游戏数据库、一个网站克隆工具(goclone)、一个抓取邮箱的命令行工具,docker-slim 也用它来优化容器。
在哪能找到可以学习的示例爬虫?Colly 仓库里的 examples 文件夹包含比 README 中基本代码片段更详细的示例——很适合用来照着写你的第一个爬虫。
为什么安装路径以 /v2 结尾?Go modules 要求库到了 v2 及以上版本必须在路径中带上主版本号后缀,所以你要安装并导入 github.com/gocolly/colly/v2 才能获得当前的主版本。
Creator
gocolly
Size
9.5 MB
Stars
25.6K
Age
9y
Commits · 90 Days