nsfw_data_scraper README

The project's README from github.com/alex000kim/nsfw_data_scraper. As one markdown file: README.md.

NSFW Data Scraper

Note: use with caution - the dataset is noisy

Description

This is a set of scripts that allows for an automatic collection of tens of thousands of images for the following (loosely defined) categories to be later used for training an image classifier:

porn - pornography images
hentai - hentai images, but also includes pornographic drawings
sexy - sexually explicit images, but not pornography. Think nude photos, playboy, bikini, etc.
neutral - safe for work neutral images of everyday things and people
drawings - safe for work drawings (including anime)

Here is what each script (located under scripts directory) does:

1_get_urls_.sh - iterates through text files under scripts/source_urls downloading URLs of images for each of the 5 categories above. The ripme application performs all the heavy lifting. The source URLs are mostly links to various subreddits, but could be any website that Ripme supports. Note: I already ran this script for you, and its outputs are located in raw_data directory. No need to rerun unless you edit files under scripts/source_urls.
2_download_from_urls_.sh - downloads actual images for urls found in text files in raw_data directory.
3_optional_download_drawings_.sh - (optional) script that downloads SFW anime images from the Danbooru2018 database.
4_optional_download_neutral_.sh - (optional) script that downloads SFW neutral images from the Caltech256 dataset
5_create_train_.sh - creates data/train directory and copy all *.jpg and *.jpeg files into it from raw_data. Also removes corrupted images.
6_create_test_.sh - creates data/test directory and moves N=2000 random files for each class from data/train to data/test (change this number inside the script if you need a different train/test split). Alternatively, you can run it multiple times, each time it will move N images for each class from data/train to data/test.

Prerequisites

Docker

How to collect data

$ docker build . -t docker_nsfw_data_scraper
Sending build context to Docker daemon  426.3MB
Step 1/3 : FROM ubuntu:18.04
 ---> 775349758637
Step 2/3 : RUN apt update  && apt upgrade -y  && apt install wget rsync imagemagick default-jre -y
 ---> Using cache
 ---> b2129908e7e2
Step 3/3 : ENTRYPOINT ["/bin/bash"]
 ---> Using cache
 ---> d32c5ae5235b
Successfully built d32c5ae5235b
Successfully tagged docker_nsfw_data_scraper:latest
$ # Next command might run for several hours. It is recommended to leave it overnight
$ docker run -v $(pwd):/root/nsfw_data_scraper docker_nsfw_data_scraper scripts/runall.sh
Getting images for class: neutral
...
...
$ ls data
test  train
$ ls data/train/
drawings  hentai  neutral  porn  sexy
$ ls data/test/
drawings  hentai  neutral  porn  sexy

How to train a CNN model

Install fastai: conda install -c pytorch -c fastai fastai
Run train_model.ipynb top to bottom

Results

I was able to train a CNN classifier to 91% accuracy with the following confusion matrix:

alt text

As expected, drawings and hentai are confused with each other more frequently than with other classes.

Same with porn and sexy categories.

Searching Database…

12.6K ·GitHub

Nsfw_data_scraper

What is this?

This is a collection of shell scripts that scrapes and aggregates tens of thousands of images across 5 categories (porn, hentai, sexy, neutral, drawings) to build a labeled dataset for training an NSFW image classifier.

Is this a trained model or just scripts?

It is not an app or a ready model — it is a set of numbered shell scripts (1_get_urls.sh through 6_create_test.sh) you run in sequence to scrape, download, and organize the raw image dataset yourself.

How do I collect the dataset?

Run the scripts under scripts/ in order: the first script grabs image URLs per category using the bundled Ripme app, the second downloads the actual images, and later scripts split everything into train/test folders.

Do I need Docker or Python installed first?

You need Docker to run the scraping pipeline, and fastai (installed via conda) if you also want to train the CNN classifier on the resulting dataset.

Is the resulting dataset clean and ready to use?

No, the author explicitly warns it's noisy, and in their own tests the drawings/hentai and porn/sexy categories were frequently confused with each other.

What are the 5 image categories?

Porn, hentai, sexy, neutral, and drawings — covering explicit content, suggestive-but-not-explicit images, and two safe-for-work categories used for contrast during training.

Where do the safe-for-work comparison images come from?

Optional scripts pull SFW anime drawings from the Danbooru2018 database and SFW neutral photos from the Caltech256 dataset.

What is Ripme and why does the project need it?

Ripme is a third-party downloader application the scripts call to actually scrape images from the source URLs, most of which point to various subreddits.

Do I have to re-scrape the URLs myself?

No — the author already ran the URL-collection script and committed its output under the rawdata directory, so you only need to rerun it if you edit the source URL lists.

What are similar projects?

gallery-dl, waifu2x, X-AnyLabeling, fiftyone, label-studio, fashion-mnist, instaloader, FreePEP

Which topics is it in?

reddit, anime, scraper, image-classification, jupyter-notebook, image, dataset, web-scraping, data-science, shell, computer-vision, command-line-tool

Creator: alex000kim
Size: 8.3 MB
Stars: 12.6K
Updated: CommitApr 17
Age: 7y
Commits · 90 Days