deeplearning-models README

The project's README from github.com/rasbt/deeplearning-models. As one markdown file: README.md.

Deep Learning Models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

Title Dataset Description Notebooks
Perceptron 2D toy data TBD PyTorch TensorFlow
Logistic Regression 2D toy data TBD PyTorch TensorFlow
Softmax Regression (Multinomial Logistic Regression) MNIST TBD PyTorch TensorFlow
Softmax Regression with MLxtend's plot_decision_regions on Iris Iris TBD PyTorch

Multilayer Perceptrons

Title Dataset Description Notebooks
Multilayer Perceptron MNIST TBD PyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Dropout MNIST TBD PyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Batch Normalization MNIST TBD PyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Backpropagation from Scratch MNIST TBD PyTorch TensorFlow

Convolutional Neural Networks

Basic

Title Dataset Description Notebooks
Convolutional Neural Network TBD TBD PyTorch Lightning PyTorch TensorFlow
CNN with He Initialization TBD TBD PyTorch Lightning PyTorch

Concepts

Title Dataset Description Notebooks
Replacing Fully-Connected by Equivalent Convolutional Layers TBD TBD PyTorch

AlexNet

Title Dataset Description Notebooks
AlexNet Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch
AlexNet with Grouped Convolutions Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch

DenseNet

Title Description Daset Notebooks
DenseNet-121 Digit Classifier Trained on MNIST TBD TBD PyTorch Lightning PyTorch
DenseNet-121 Image Classifier Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch

Fully Convolutional

Title Dataset Description Notebooks
"All Convolutionl Net" -- A Fully Convolutional Neural Network TBD TBD PyTorch Lightning PyTorch

LeNet

Title Dataset Description Notebooks
LeNet-5 on MNIST TBD TBD PyTorch Lightning PyTorch
LeNet-5 on CIFAR-10 TBD TBD PyTorch Lightning PyTorch
LeNet-5 on QuickDraw TBD TBD PyTorch Lightning PyTorch

MobileNet

Title Dataset Description Notebooks
MobileNet-v2 on Cifar-10 TBD TBD PyTorch Lightning PyTorch
MobileNet-v3 small on Cifar-10 TBD TBD PyTorch Lightning PyTorch
MobileNet-v3 large on Cifar-10 TBD TBD PyTorch Lightning PyTorch
MobileNet-v3 large on MNIST via Embetter TBD TBD PyTorch

Network in Network

Title Dataset Description Notebooks
Network in Network Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch

VGG

Title Dataset Description Notebooks
Convolutional Neural Network VGG-16 Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch TensorFlow
VGG-16 Smile Classifier CelebA TBD PyTorch Lightning PyTorch
VGG-16 Dogs vs Cats Classifier TBD TBD PyTorch
Convolutional Neural Network VGG-19 TBD TBD PyTorch Lightning PyTorch

ResNet

Title Dataset Description Notebooks
ResNet and Residual Blocks MNIST TBD PyTorch
ResNet-18 Digit Classifier MNIST TBD PyTorch
ResNet-18 Gender Classifier CelebA TBD PyTorch
ResNet-34 Digit Classifier MNIST TBD PyTorch
ResNet-34 Object Classifier QuickDraw TBD PyTorch
ResNet-34 Gender Classifier CelebA TBD PyTorch
ResNet-50 Digit Classifier MNIST TBD PyTorch
ResNet-50 Gender Classifier CelebA TBD PyTorch
ResNet-101 Gender Classifier CelebA TBD PyTorch
ResNet-101 CIFAR-10 TBD PyTorch
ResNet-152 Gender Classifier CelebA TBD PyTorch

Transformers

Title Dataset Description Notebooks
Multilabel DistilBERT Jigsaw Toxic Comment Challenge DistilBERT classifier fine-tuning PyTorch
DistilBERT as feature extractor IMDB movie review DistilBERT classifier with sklearn random forest and logistic regression PyTorch
DistilBERT as feature extractor using embetter IMDB movie review DistilBERT classifier with sklearn random forest and logistic regression using the scikit-learn embetter library PyTorch
Fine-tune DistilBERT I IMDB movie review Fine-tune only the last 2 layers of DistilBERT classifier PyTorch Lightning
Fine-tune DistilBERT II IMDB movie review Fine-tune the whole DistilBERT classifier PyTorch PyTorch Lightning

Ordinal Regression and Deep Learning

Please note that the following notebooks below provide reference implementations to use the respective methods. They are not performance benchmarks.

Title Dataset Description Notebooks
Baseline multilayer perceptron Cement A baseline multilayer perceptron for classification trained with the standard cross entropy loss PyTorch PyTorch Lightning
CORAL multilayer perceptron Cement Implementation of Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation 2020 PyTorch PyTorch Lightning
CORN multilayer perceptron Cement Implementation of Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities 2022 PyTorch PyTorch Lightning
Binary extension multilayer perceptron Cement Implementation of Ordinal Regression with Multiple Output CNN for Age Estimation 2016 PyTorch PyTorch Lightning
Reformulated squared-error multilayer perceptron Cement Implementation of A simple squared-error reformulation for ordinal classification 2016 PyTorch PyTorch Lightning
Class distance weighted cross-entropy loss Cement Implementation of Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation 2022 PyTorch PyTorch Lightning

Normalization Layers

Title Dataset Description Notebooks
BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier TBD TBD PyTorch
Filter Response Normalization for Network-in-Network CIFAR-10 Classifier TBD TBD PyTorch

Metric Learning

Title Dataset Description Notebooks
Siamese Network with Multilayer Perceptrons TBD TBD TensorFlow

Autoencoders

Fully-connected Autoencoders

Title Dataset Description Notebooks
Autoencoder (MNIST) TBD TBD PyTorch TensorFlow
Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier TBD TBD PyTorch TensorFlow

Convolutional Autoencoders

Title Dataset Description Notebooks
Convolutional Autoencoder with Deconvolutions / Transposed Convolutions TBD TBD PyTorch TensorFlow
Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance TBD TBD PyTorch
Convolutional Autoencoder with Deconvolutions (without pooling operations) TBD TBD PyTorch
Convolutional Autoencoder with Nearest-neighbor Interpolation TBD TBD PyTorch TensorFlow
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA TBD TBD PyTorch
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw TBD TBD PyTorch

Variational Autoencoders

Title Dataset Description Notebooks
Variational Autoencoder TBD TBD PyTorch
Convolutional Variational Autoencoder TBD TBD PyTorch

Conditional Variational Autoencoders

Title Dataset Description Notebooks
Conditional Variational Autoencoder (with labels in reconstruction loss) TBD TBD PyTorch
Conditional Variational Autoencoder (without labels in reconstruction loss) TBD TBD PyTorch
Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss) TBD TBD PyTorch
Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss) TBD TBD PyTorch

Generative Adversarial Networks (GANs)

Title Dataset Description Notebooks
Fully Connected GAN on MNIST TBD TBD PyTorch TensorFlow
Fully Connected Wasserstein GAN on MNIST TBD TBD PyTorch
Convolutional GAN on MNIST TBD TBD PyTorch TensorFlow
Convolutional GAN on MNIST with Label Smoothing TBD TBD PyTorch TensorFlow
Convolutional Wasserstein GAN on MNIST TBD TBD PyTorch
Deep Convolutional GAN (DCGAN) on Cats and Dogs Images TBD TBD PyTorch
Deep Convolutional GAN (DCGAN) on CelebA Face Images TBD TBD PyTorch

Graph Neural Networks (GNNs)

Title Dataset Description Notebooks
Most Basic Graph Neural Network with Gaussian Filter on MNIST TBD TBD PyTorch
Basic Graph Neural Network with Edge Prediction on MNIST TBD TBD PyTorch
Basic Graph Neural Network with Spectral Graph Convolution on MNIST TBD TBD PyTorch

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

Title Dataset Description Notebooks
A simple single-layer RNN (IMDB) TBD TBD PyTorch
A simple single-layer RNN with packed sequences to ignore padding characters (IMDB) TBD TBD PyTorch
RNN with LSTM cells (IMDB) TBD TBD PyTorch
RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors TBD TBD PyTorch
RNN with LSTM cells and Own Dataset in CSV Format (IMDB) TBD TBD PyTorch
RNN with GRU cells (IMDB) TBD TBD PyTorch
Multilayer bi-directional RNN (IMDB) TBD TBD PyTorch
Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News) TBD TBD PyTorch

Many-to-Many / Sequence-to-Sequence

Title Dataset Description Notebooks
A simple character RNN to generate new text (Charles Dickens) TBD TBD PyTorch

Model Evaluation

K-Fold Cross-Validation

Title Dataset Description Notebooks
Baseline CNN MNIST A simple baseline with traditional train/validation/test splits PyTorch PyTorch Lightning
K-fold with pl_cross MNIST A 5-fold cross-validation run using the pl_cross library PyTorch Lightning

Data Augmentation

Title Dataset Description Notebooks
AutoAugment & TrivialAugment for Image Data CIFAR-10 Trains a ResNet-18 using AutoAugment and TrivialAugment PyTorch Lightning

Tips and Tricks

Title Dataset Description Notebooks
Cyclical Learning Rate TBD TBD PyTorch
Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet) TBD TBD PyTorch
Gradient Clipping (w. MLP on MNIST) TBD TBD PyTorch

Transfer Learning

Title Dataset Description Notebooks
Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10) TBD TBD PyTorch

Visualization and Interpretation

Title Dataset Description Notebooks
Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) TBD TBD PyTorch
Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) TBD TBD PyTorch

PyTorch Workflows and Mechanics

PyTorch Lightning Examples

Title Dataset Description Notebooks
MLP in Lightning with TensorBoard -- continue training the last model TBD TBD PyTorch
MLP in Lightning with TensorBoard -- checkpointing best model TBD TBD PyTorch

Custom Datasets

Title Dataset Description Notebooks
Custom Data Loader Example for PNG Files TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5 TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD) TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST TBD TBD PyTorch

Training and Preprocessing

Title Dataset Description Notebooks
PyTorch DataLoader State and Nested Iterations Toy Explains DataLoader behavior when in nested functions PyTorch
Generating Validation Set Splits TBD TBD PyTorch
Dataloading with Pinned Memory TBD TBD PyTorch
Standardizing Images TBD TBD PyTorch
Image Transformation Examples TBD TBD PyTorch
Char-RNN with Own Text File TBD TBD PyTorch
Sentiment Classification RNN with Own CSV File TBD TBD PyTorch

Improving Memory Efficiency

Title Dataset Description Notebooks
Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10) TBD TBD PyTorch

Parallel Computing

Title Description Notebooks
Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA TBD PyTorch
Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example) TBD PyTorch

Other

Title Dataset Description Notebooks
PyTorch with and without Deterministic Behavior -- Runtime Benchmark TBD TBD PyTorch
Sequential API and hooks TBD TBD PyTorch
Weight Sharing Within a Layer TBD TBD PyTorch
Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib TBD TBD PyTorch

Autograd

Title Dataset Description Notebooks
Getting Gradients of an Intermediate Variable in PyTorch TBD TBD PyTorch

TensorFlow Workflows and Mechanics

Custom Datasets

Title Description Notebooks
Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives TBD TensorFlow
Storing an Image Dataset for Minibatch Training using HDF5 TBD TensorFlow
Using Input Pipelines to Read Data from TFRecords Files TBD TensorFlow
Using Queue Runners to Feed Images Directly from Disk TBD TensorFlow
Using TensorFlow's Dataset API TBD TensorFlow

Training and Preprocessing

Title Dataset Description Notebooks
Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives TBD TBD TensorFlow
Title Description Notebooks
TorchMetrics How do we use it, and what's the difference between .update() and .forward()? PyTorch

deeplearning-models (rasbt/deeplearning-models)

Searching Database…

这是什么?这是一个大型 Jupyter Notebook 合集,实现了多种深度学习架构和模型,并提供基于 TensorFlow 和 PyTorch 的训练技巧。
这是要导入的库,还是一组示例笔记本?它是一个独立 Jupyter 笔记本的参考合集,而不是可以安装的库。每个笔记本都是一份自包含的实现,供你阅读和运行,而不是需要导入的包。
这些笔记本怎么用?克隆仓库,然后在 Jupyter 中打开你想要的模型对应的笔记本。每个笔记本都可以单独运行,通常使用 MNIST、CIFAR-10 或 CelebA 等标准数据集进行训练。
我需要同时安装 TensorFlow 和 PyTorch 吗?你只需要安装某个笔记本所用的框架即可。这个合集同时包含 TensorFlow 和 PyTorch 的笔记本,所需依赖取决于你打开的具体笔记本。
所有笔记本都已经完整填写并有文档说明了吗?没有。很多表格条目中的数据集、描述和笔记本链接仍标注为占位符(TBD),因此各条目的完善程度和文档质量参差不齐。
这个合集是给谁看的?面向希望参考经典和现代深度学习架构实现(CNN、RNN、GAN、VAE、GNN、Transformer)以便学习或改编的学生和从业者。
合集涵盖了哪些模型系列?涵盖传统机器学习(感知机、逻辑回归/Softmax 回归)、多层感知机(MLP)、CNN(AlexNet、VGG、ResNet、DenseNet、MobileNet、LeNet)、Transformer(DistilBERT)、RNN、自编码器、VAE、GAN 以及图神经网络(GNN)。
序数回归笔记本是用来做性能基准测试的吗?不是。它们明确是对 CORAL 和 CORN 等序数回归论文方法的参考实现,而不是性能基准测试。
是否包含 PyTorch Lightning 的示例?包含。有几个 PyTorch Lightning 笔记本,演示了检查点保存,以及如何结合 TensorBoard 日志继续训练。
这些笔记本中常用哪些数据集?最常用的数据集有 MNIST、CIFAR-10、CelebA、IMDB 影评和 QuickDraw。
Creator
rasbt
Size
43.1 MB
Stars
17.6K
Updated
CommitSep 26
Age
7y
Commits · 90 Days