nsfw_data_scraper README

The project's README from github.com/alex000kim/nsfw_data_scraper. As one markdown file: README.md.

NSFW Data Scraper

Note: use with caution - the dataset is noisy

Description

This is a set of scripts that allows for an automatic collection of tens of thousands of images for the following (loosely defined) categories to be later used for training an image classifier:

porn - pornography images
hentai - hentai images, but also includes pornographic drawings
sexy - sexually explicit images, but not pornography. Think nude photos, playboy, bikini, etc.
neutral - safe for work neutral images of everyday things and people
drawings - safe for work drawings (including anime)

Here is what each script (located under scripts directory) does:

1_get_urls_.sh - iterates through text files under scripts/source_urls downloading URLs of images for each of the 5 categories above. The ripme application performs all the heavy lifting. The source URLs are mostly links to various subreddits, but could be any website that Ripme supports. Note: I already ran this script for you, and its outputs are located in raw_data directory. No need to rerun unless you edit files under scripts/source_urls.
2_download_from_urls_.sh - downloads actual images for urls found in text files in raw_data directory.
3_optional_download_drawings_.sh - (optional) script that downloads SFW anime images from the Danbooru2018 database.
4_optional_download_neutral_.sh - (optional) script that downloads SFW neutral images from the Caltech256 dataset
5_create_train_.sh - creates data/train directory and copy all *.jpg and *.jpeg files into it from raw_data. Also removes corrupted images.
6_create_test_.sh - creates data/test directory and moves N=2000 random files for each class from data/train to data/test (change this number inside the script if you need a different train/test split). Alternatively, you can run it multiple times, each time it will move N images for each class from data/train to data/test.

Prerequisites

Docker

How to collect data

$ docker build . -t docker_nsfw_data_scraper
Sending build context to Docker daemon  426.3MB
Step 1/3 : FROM ubuntu:18.04
 ---> 775349758637
Step 2/3 : RUN apt update  && apt upgrade -y  && apt install wget rsync imagemagick default-jre -y
 ---> Using cache
 ---> b2129908e7e2
Step 3/3 : ENTRYPOINT ["/bin/bash"]
 ---> Using cache
 ---> d32c5ae5235b
Successfully built d32c5ae5235b
Successfully tagged docker_nsfw_data_scraper:latest
$ # Next command might run for several hours. It is recommended to leave it overnight
$ docker run -v $(pwd):/root/nsfw_data_scraper docker_nsfw_data_scraper scripts/runall.sh
Getting images for class: neutral
...
...
$ ls data
test  train
$ ls data/train/
drawings  hentai  neutral  porn  sexy
$ ls data/test/
drawings  hentai  neutral  porn  sexy

How to train a CNN model

Install fastai: conda install -c pytorch -c fastai fastai
Run train_model.ipynb top to bottom

Results

I was able to train a CNN classifier to 91% accuracy with the following confusion matrix:

alt text

As expected, drawings and hentai are confused with each other more frequently than with other classes.

Same with porn and sexy categories.

Searching Database…

12.6K ·GitHub

Nsfw_data_scraper

这是什么？

这是一组 shell 脚本，用于抓取并汇总 5 个类别（色情、成人动漫、性感、中性、绘画）中的数万张图片，构建用于训练 NSFW 图片分类器的带标签数据集。

这是训练好的模型，还是只是脚本？

它既不是应用程序，也不是现成的模型，而是一组编号的 shell 脚本（从 1_get_urls.sh 到 6_create_test.sh），需要你按顺序运行，自行抓取、下载并整理原始图片数据集。

我该如何收集数据集？

按顺序运行 scripts/ 目录下的脚本：第一个脚本通过随附的 Ripme 应用按类别抓取图片网址，第二个脚本下载实际图片，之后的脚本再把所有内容拆分到训练和测试文件夹中。

使用前我需要先安装 Docker 或 Python 吗？

运行抓取流程需要 Docker；如果你还想在生成的数据集上训练 CNN 分类器，则需要通过 conda 安装 fastai。

生成的数据集干净、可以直接使用吗？

不行。作者明确提醒数据噪声较多，而且在他自己的测试中，绘画与成人动漫、色情与性感这两组类别经常被混淆。

5 个图片类别分别是什么？

色情、成人动漫、性感、中性和绘画。前三类分别涵盖露骨内容和暗示性但不露骨的图片，后两类是适合工作场合查看的类别，在训练时用作对比。

用于对比的适合工作场合（SFW）图片从哪里来？

可选脚本会从 Danbooru2018 数据库中抓取适合工作场合的动漫绘画，并从 Caltech256 数据集中抓取适合工作场合的中性照片。

什么是 Ripme，项目为什么需要它？

Ripme 是第三方下载应用，脚本会调用它从源网址实际抓取图片，这些网址大多指向各种 Reddit 子版块。

我需要自己重新抓取网址吗？

不需要。作者已经运行过网址收集脚本，并把输出结果提交到了 rawdata 目录下，因此只有在你修改了源网址列表时才需要重新运行。

What are similar projects?

gallery-dl, waifu2x, X-AnyLabeling, fiftyone, label-studio, fashion-mnist, instaloader, FreePEP

Which topics is it in?

reddit, anime, scraper, image-classification, jupyter-notebook, image, dataset, web-scraping, data-science, shell, computer-vision, command-line-tool

Creator: alex000kim
Size: 8.3 MB
Stars: 12.6K
Updated: CommitApr 17
Age: 7y
Commits · 90 Days