22.2K ·GitHub
What is this?
A free Chinese-language tutorial series that teaches Python web scraping from zero — 30+ numbered lessons and mitmproxy guides, from first packet capture to distributed crawling, with source code for the practice projects.
A free Chinese-language tutorial series that teaches Python web scraping from zero — 30+ numbered lessons and mitmproxy guides, from first packet capture to distributed crawling, with source code for the practice projects.Is this a tool I install, or a course I read through?It's a course, not a tool: the README indexes lessons published as articles, and the repo holds the matching source code. Nothing to install — you read and code along.
It's a course, not a tool: the README indexes lessons published as articles, and the repo holds the matching source code. Nothing to install — you read and code along.How do I work through it — is there a set order?Read top to bottom: it starts with a pre-basics section and packet capture, then numbered lessons 00-30 that build on each other, and each hands-on project links to its source code.
Read top to bottom: it starts with a pre-basics section and packet capture, then numbered lessons 00-30 that build on each other, and each hands-on project links to its source code.What do I need to have installed to follow along?Python 3 to start. Lessons then bring in Chrome, Fiddler or mitmproxy for packet capture, selenium for browser automation, Appium plus a phone for app scraping, and MySQL or MongoDB for storage.
Python 3 to start. Lessons then bring in Chrome, Fiddler or mitmproxy for packet capture, selenium for browser automation, Appium plus a phone for app scraping, and MySQL or MongoDB for storage.Is the series complete, or still growing?It ends with 'to be continued' (未完待续): the numbering jumps from lesson 28 to 30 and there is no finale. The repo dates to 2019, so a few targeted sites and tools have aged.
It ends with 'to be continued' (未完待续): the numbering jumps from lesson 28 to 30 and there is no finale. The repo dates to 2019, so a few targeted sites and tools have aged.I've never coded before — can I start here?Yes — the first lesson targets people who can't code at all, then it ramps up to JS reverse engineering and distributed scraping. Friendly entry for beginners, advanced by the end.
Yes — the first lesson targets people who can't code at all, then it ramps up to JS reverse engineering and distributed scraping. Friendly entry for beginners, advanced by the end.How does it deal with anti-scraping — captchas, IP bans, logins?With dedicated lessons: faking request headers, rotating IP proxy pools, three ways into login-required sites, recognizing image captchas, auto-solving Bilibili's slider captcha, plus defeating CSS font encryption and JS obfuscation.
With dedicated lessons: faking request headers, rotating IP proxy pools, three ways into login-required sites, recognizing image captchas, auto-solving Bilibili's slider captcha, plus defeating CSS font encryption and JS obfuscation.Can it scrape mobile apps, not just websites?Yes — it covers Fiddler phone capture, mitmproxy to monitor your phone's traffic, and Appium lessons that drive a phone to scrape WeChat Moments, even unpacking an Android app's low-level encryption.
Yes — it covers Fiddler phone capture, mitmproxy to monitor your phone's traffic, and Appium lessons that drive a phone to scrape WeChat Moments, even unpacking an Android app's low-level encryption.Are there complete example projects with code I can study?Ten, with linked source: Dangdang top-500 books, Douban top-250 movies, Bilibili scraping, a slider-captcha solver, WeChat Moments via Appium, Qiushibaike jokes into MongoDB, 200k emoji stickers, and WeChat articles saved as PDF.
Ten, with linked source: Dangdang top-500 books, Douban top-250 movies, Bilibili scraping, a slider-captcha solver, WeChat Moments via Appium, Qiushibaike jokes into MongoDB, 200k emoji stickers, and WeChat articles saved as PDF.What are similar projects?python-spider, Python, awesome-python-login-model, examples-of-web-crawlers, botasaurus, Python-100-Days, TinyWebServer, undetected-chromedriver
python-spider, Python, awesome-python-login-model, examples-of-web-crawlers, botasaurus, Python-100-Days, TinyWebServer, undetected-chromedriverWhich topics is it in?mobile-automation, selenium, mitm, captcha, distributed, multithreading, crawler, python3, concurrency, mongodb, wechat, ocr
mobile-automation, selenium, mitm, captcha, distributed, multithreading, crawler, python3, concurrency, mongodb, wechat, ocr