DeepLearningExamples README

The project's README from github.com/NVIDIA/DeepLearningExamples. As one markdown file: README.md.

NVIDIA Deep Learning Examples for Tensor Cores

Introduction

This repository provides State-of-the-Art Deep Learning examples that are easy to train and deploy, achieving the best reproducible accuracy and performance with NVIDIA CUDA-X software stack running on NVIDIA Volta, Turing and Ampere GPUs.

NVIDIA GPU Cloud (NGC) Container Registry

These examples, along with our NVIDIA deep learning software stack, are provided in a monthly updated Docker container on the NGC container registry (https://ngc.nvidia.com). These containers include:

Computer Vision

Models Framework AMP Multi-GPU Multi-Node TensorRT ONNX Triton DLC NB
EfficientNet-B0 PyTorch Yes Yes - Supported - Supported Yes -
EfficientNet-B4 PyTorch Yes Yes - Supported - Supported Yes -
EfficientNet-WideSE-B0 PyTorch Yes Yes - Supported - Supported Yes -
EfficientNet-WideSE-B4 PyTorch Yes Yes - Supported - Supported Yes -
EfficientNet v1-B0 TensorFlow2 Yes Yes Yes Example - Supported Yes -
EfficientNet v1-B4 TensorFlow2 Yes Yes Yes Example - Supported Yes -
EfficientNet v2-S TensorFlow2 Yes Yes Yes Example - Supported Yes -
GPUNet PyTorch Yes Yes - Example Yes Example Yes -
Mask R-CNN PyTorch Yes Yes - Example - Supported - Yes
Mask R-CNN TensorFlow2 Yes Yes - Example - Supported Yes -
nnUNet PyTorch Yes Yes - Supported - Supported Yes -
ResNet-50 MXNet Yes Yes - Supported - Supported - -
ResNet-50 PaddlePaddle Yes Yes - Example - Supported - -
ResNet-50 PyTorch Yes Yes - Example - Example Yes -
ResNet-50 TensorFlow Yes Yes - Supported - Supported Yes -
ResNeXt-101 PyTorch Yes Yes - Example - Example Yes -
ResNeXt-101 TensorFlow Yes Yes - Supported - Supported Yes -
SE-ResNeXt-101 PyTorch Yes Yes - Example - Example Yes -
SE-ResNeXt-101 TensorFlow Yes Yes - Supported - Supported Yes -
SSD PyTorch Yes Yes - Supported - Supported - Yes
SSD TensorFlow Yes Yes - Supported - Supported Yes Yes
U-Net Med TensorFlow2 Yes Yes - Example - Supported Yes -

Natural Language Processing

Models Framework AMP Multi-GPU Multi-Node TensorRT ONNX Triton DLC NB
BERT PyTorch Yes Yes Yes Example - Example Yes -
GNMT PyTorch Yes Yes - Supported - Supported - -
ELECTRA TensorFlow2 Yes Yes Yes Supported - Supported Yes -
BERT TensorFlow Yes Yes Yes Example - Example Yes Yes
BERT TensorFlow2 Yes Yes Yes Supported - Supported Yes -
GNMT TensorFlow Yes Yes - Supported - Supported - -
Faster Transformer Tensorflow - - - Example - Supported - -

Recommender Systems

Models Framework AMP Multi-GPU Multi-Node ONNX Triton DLC NB
DLRM PyTorch Yes Yes - Yes Example Yes Yes
DLRM TensorFlow2 Yes Yes Yes - Supported Yes -
NCF PyTorch Yes Yes - - Supported - -
Wide&Deep TensorFlow Yes Yes - - Supported Yes -
Wide&Deep TensorFlow2 Yes Yes - - Supported Yes -
NCF TensorFlow Yes Yes - - Supported Yes -
VAE-CF TensorFlow Yes Yes - - Supported - -
SIM TensorFlow2 Yes Yes - - Supported Yes -

Speech to Text

Models Framework AMP Multi-GPU Multi-Node TensorRT ONNX Triton DLC NB
Jasper PyTorch Yes Yes - Example Yes Example Yes Yes
QuartzNet PyTorch Yes Yes - Supported - Supported Yes -

Text to Speech

Models Framework AMP Multi-GPU Multi-Node TensorRT ONNX Triton DLC NB
FastPitch PyTorch Yes Yes - Example - Example Yes Yes
FastSpeech PyTorch Yes Yes - Example - Supported - -
Tacotron 2 and WaveGlow PyTorch Yes Yes - Example Yes Example Yes -
HiFi-GAN PyTorch Yes Yes - Supported - Supported Yes -

Graph Neural Networks

Models Framework AMP Multi-GPU Multi-Node ONNX Triton DLC NB
SE(3)-Transformer PyTorch Yes Yes - - Supported - -
MoFlow PyTorch Yes Yes - - Supported - -

Time-Series Forecasting

Models Framework AMP Multi-GPU Multi-Node TensorRT ONNX Triton DLC NB
Temporal Fusion Transformer PyTorch Yes Yes - Example Yes Example Yes -

NVIDIA support

In each of the network READMEs, we indicate the level of support that will be provided. The range is from ongoing updates and improvements to a point-in-time release for thought leadership.

Glossary

Multinode Training Supported on a pyxis/enroot Slurm cluster.

Deep Learning Compiler (DLC) TensorFlow XLA and PyTorch JIT and/or TorchScript

Accelerated Linear Algebra (XLA) XLA is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes. The results are improvements in speed and memory usage.

PyTorch JIT and/or TorchScript TorchScript is a way to create serializable and optimizable models from PyTorch code. TorchScript, an intermediate representation of a PyTorch model (subclass of nn.Module) that can then be run in a high-performance environment such as C++.

Automatic Mixed Precision (AMP) Automatic Mixed Precision (AMP) enables mixed precision training on Volta, Turing, and NVIDIA Ampere GPU architectures automatically.

TensorFloat-32 (TF32) TensorFloat-32 (TF32) is the new math mode in NVIDIA A100 GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. TF32 is supported in the NVIDIA Ampere GPU architecture and is enabled by default.

Jupyter Notebooks (NB) The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

Feedback / Contributions

We're posting these examples on GitHub to better support the community, facilitate feedback, as well as collect and implement contributions using GitHub Issues and pull requests. We welcome all contributions!

Known issues

In each of the network READMEs, we indicate any known issues and encourage the community to provide feedback.

DeepLearningExamples (NVIDIA/DeepLearningExamples)

Searching Database…

这是什么?由 NVIDIA 维护的一组最先进的深度学习参考实现,涵盖计算机视觉、自然语言处理、推荐系统、语音等领域,并针对 NVIDIA Tensor Core GPU 进行调优,以保证结果可复现、性能出色。
这是一个库,还是一堆独立的模型项目?它是许多独立模型实现的索引(如 ResNet、BERT、DLRM、Tacotron 2 等)。每个模型各占一个文件夹,并附有自己的 README,分别基于 PyTorch、TensorFlow、TensorFlow2、MXNet 和 PaddlePaddle 实现。
我到底该怎么运行其中一个示例?打开你想用的那个模型的 README,按其中的说明操作;也可以从 NGC 仓库拉取 NVIDIA 每月更新的 Docker 容器,里面打包了这些示例以及完整的优化软件环境。
使用这些示例是否必须有特定的 NVIDIA GPU?是的。这些示例是针对 NVIDIA Volta、Turing 和 Ampere 架构的 GPU 构建和调优的,以便充分利用 Tensor Core、AMP 和 TF32。
所有模型的生产就绪程度和支持情况都一样吗?不是。各个模型和功能的支持级别各不相同:有的模型获得了 TensorRT/Triton 的完整“Supported”(官方支持)状态,有的只有“Example”(示例)级别;多节点训练仅限少数模型。每个模型的 README 都会说明其自身的支持等级。
README 里提到的 NGC 容器是什么?它是 NVIDIA GPU Cloud 提供的每月更新的 Docker 镜像,内含这些示例、NVIDIA 对上游框架的贡献,以及 cuDNN、NCCL、cuBLAS 等经过性能质检的优化库。
AMP 和 TF32 对这些模型意味着什么?AMP(自动混合精度)和 TF32 是 Volta、Turing、Ampere GPU 上的精度模式,几乎无需改动代码就能加快训练速度(在 Volta 上,TF32 相比 FP32 最高可提速 10 倍),并且已在大多数列出的模型中启用。
这些示例涵盖哪些问题领域?计算机视觉(分类、检测、分割)、自然语言处理(BERT、GNMT、ELECTRA)、推荐系统(DLRM、NCF、Wide&Deep)、语音转文字(Jasper、QuartzNet)、文字转语音(Tacotron 2、HiFi-GAN)、图神经网络,以及时间序列预测(Temporal Fusion Transformer)。
仓库里的每个模型都在持续更新吗?不一定。每个模型的 README 都会注明其支持等级,从持续更新和改进,到仅作为一次性的“point-in-time”版本发布、主要用于展示技术引领作用,各不相同。
Creator
NVIDIA
Size
106.2 MB
Stars
14.9K
Updated
CommitApr 04
Age
8y
Commits · 90 Days