What is this?
A collection of Python 3 example scripts that simulate logging into popular websites — mostly Chinese ones like Zhihu, Bilibili, Weibo, JD (JingDong) and Taobao, plus Facebook and GitHub — together with a few simple crawlers. It is meant as a learning resource for login simulation and scraping techniques.
A collection of Python 3 example scripts that simulate logging into popular websites — mostly Chinese ones like Zhihu, Bilibili, Weibo, JD (JingDong) and Taobao, plus Facebook and GitHub — together with a few simple crawlers. It is meant as a learning resource for login simulation and scraping techniques.Is this one program I install, or a pile of separate scripts?A pile of separate scripts: each site gets its own standalone example written in Python 3, and you run only the one you need. There is no package to install and no single CLI that drives every site.
A pile of separate scripts: each site gets its own standalone example written in Python 3, and you run only the one you need. There is no package to install and no single CLI that drives every site.How do I run one of these login scripts?The README has no install steps; in practice you clone the repo, pick the script for your target site, set up selenium with a Chrome or Firefox WebDriver, and run it with Python 3.
The README has no install steps; in practice you clone the repo, pick the script for your target site, set up selenium with a Chrome or Firefox WebDriver, and run it with Python 3.What do I need installed besides Python?selenium and a WebDriver for Chrome or Firefox — the WebDriver section of the README names exactly those two browsers. Some scripts also assume you will reuse the login cookies with requests or scrapy for the actual data collection.
selenium and a WebDriver for Chrome or Firefox — the WebDriver section of the README names exactly those two browsers. Some scripts also assume you will reuse the login cookies with requests or scrapy for the actual data collection.Do these login methods cope with captchas?No — none of the methods in this project handle captchas. The author says recognizing complex captchas is still very hard, and his advice is to steer clear of them when scraping rather than try to solve them.
No — none of the methods in this project handle captchas. The author says recognizing complex captchas is still very hard, and his advice is to steer clear of them when scraping rather than try to solve them.Will this run on Windows, or is it Linux-only?No operating system is mentioned anywhere: these are plain Python 3 and selenium scripts for Chrome or Firefox, so any desktop OS that runs those — Windows, macOS or Linux — should work.
No operating system is mentioned anywhere: these are plain Python 3 and selenium scripts for Chrome or Firefox, so any desktop OS that runs those — Windows, macOS or Linux — should work.Is the code under an open-source license I can reuse?No license is declared in the README; the only note is that the project is intended for study and exchange, so reuse terms are undefined — do not assume standard open-source permissions.
No license is declared in the README; the only note is that the project is intended for study and exchange, so reuse terms are undefined — do not assume standard open-source permissions.Which sites are covered?Login scripts for Xiami Music, Facebook, Weibo web, Zhihu, QQ Zone, CSDN, Taobao, Baidu, Guokr, JingDong (including auto-applying for JD free trials), 163 Mail, Lagou, Bilibili, Douban, Liepin, WeChat web with friends-list grabbing, GitHub and NetEase Cloud Music, plus crawlers for Tuchong photos, QiuShi BaiKe, Baidu Tieba and Baidu Translate.
Login scripts for Xiami Music, Facebook, Weibo web, Zhihu, QQ Zone, CSDN, Taobao, Baidu, Guokr, JingDong (including auto-applying for JD free trials), 163 Mail, Lagou, Bilibili, Douban, Liepin, WeChat web with friends-list grabbing, GitHub and NetEase Cloud Music, plus crawlers for Tuchong photos, QiuShi BaiKe, Baidu Tieba and Baidu Translate.How does a login actually work — like a real user in a browser?Two ways: direct login by replaying the site's login requests (sometimes derived from JS reverse engineering), or selenium with WebDriver driving a real browser. The author notes direct login is very hard for sites like QQ Zone and Bilibili, where selenium is the easier route; after login the cookies are kept so requests or scrapy can collect data at full speed.
Two ways: direct login by replaying the site's login requests (sometimes derived from JS reverse engineering), or selenium with WebDriver driving a real browser. The author notes direct login is very hard for sites like QQ Zone and Bilibili, where selenium is the easier route; after login the cookies are kept so requests or scrapy can collect data at full speed.Are these scripts still working today?Not all of them. The catalog itself marks the Lagou login as expired and the Taobao crawler as being reworked, and the author explains that sites change their policies and styles, which breaks the code — he asks users to report issues and submit PRs with fixes. Expect to debug before a script works.
Not all of them. The catalog itself marks the Lagou login as expired and the Taobao crawler as being reworked, and the author explains that sites change their policies and styles, which breaks the code — he asks users to report issues and submit PRs with fixes. Expect to debug before a script works.Is this repo meant for learning or for real production scraping?For learning. The README states the project is for study and exchange only, and the author mentions preparing a free crawler tutorial that he claims will teach you to scrape most sites on the web. Treat every script as a worked example of a technique, not as production-ready tooling.
For learning. The README states the project is for study and exchange only, and the author mentions preparing a free crawler tutorial that he claims will teach you to scrape most sites on the web. Treat every script as a worked example of a technique, not as production-ready tooling.What are similar projects?learn_python3_spider, botasaurus, undetected-chromedriver, MediaCrawler, catbus, invisible_selenium, CloakBrowser, EasySpider
learn_python3_spider, botasaurus, undetected-chromedriver, MediaCrawler, catbus, invisible_selenium, CloakBrowser, EasySpiderWhich topics is it in?weibo, webdriver, login, selenium, captcha, crawler, web-scraping, browser-automation, reverse-engineering, chinese, python, windows
weibo, webdriver, login, selenium, captcha, crawler, web-scraping, browser-automation, reverse-engineering, chinese, python, windows