Transformers-Tutorials README

Transformers-Tutorials

Hi there!

This repository contains demos I made with the Transformers library by 🤗 HuggingFace. Currently, all of them are implemented in PyTorch.

NOTE: if you are not familiar with HuggingFace and/or Transformers, I highly recommend to check out our free course, which introduces you to several Transformer architectures (such as BERT, GPT-2, T5, BART, etc.), as well as an overview of the HuggingFace libraries, including Transformers, Tokenizers, Datasets, Accelerate and the hub.

For an overview of the ecosystem of HuggingFace for computer vision (June 2022), refer to this notebook with corresponding video.

Currently, it contains the following demos:

Audio Spectrogram Transformer (paper):
- performing inference with ASTForAudioClassification to classify audio.
BERT (paper):
- fine-tuning BertForTokenClassification on a named entity recognition (NER) dataset.
- fine-tuning BertForSequenceClassification for multi-label text classification.
BEiT (paper):
- understanding BeitForMaskedImageModeling
CANINE (paper):
- fine-tuning CanineForSequenceClassification on IMDb
CLIPSeg (paper):
- performing zero-shot image segmentation with CLIPSeg
Conditional DETR (paper):
- performing inference with ConditionalDetrForObjectDetection
- fine-tuning ConditionalDetrForObjectDetection on a custom dataset (balloon)
ConvNeXT (paper):
- fine-tuning (and performing inference with) ConvNextForImageClassification
DINO (paper):
- visualize self-attention of Vision Transformers trained using the DINO method
DETR (paper):
- performing inference with DetrForObjectDetection
- fine-tuning DetrForObjectDetection on a custom object detection dataset
- evaluating DetrForObjectDetection on the COCO detection 2017 validation set
- performing inference with DetrForSegmentation
- fine-tuning DetrForSegmentation on COCO panoptic 2017
DPT (paper):
- performing inference with DPT for monocular depth estimation
- performing inference with DPT for semantic segmentation
Deformable DETR (paper):
- performing inference with DeformableDetrForObjectDetection
DiT (paper):
- performing inference with DiT for document image classification
Donut (paper):
- performing inference with Donut for document image classification
- fine-tuning Donut for document image classification
- performing inference with Donut for document visual question answering (DocVQA)
- performing inference with Donut for document parsing
- fine-tuning Donut for document parsing with PyTorch Lightning
GIT (paper):
- performing inference with GIT for image/video captioning and image/video question-answering
- fine-tuning GIT on a custom image captioning dataset
GLPN (paper):
- performing inference with GLPNForDepthEstimation to illustrate monocular depth estimation
GPT-J-6B (repository):
- performing inference with GPTJForCausalLM to illustrate few-shot learning and code generation
GroupViT (repository):
- performing inference with GroupViTModel to illustrate zero-shot semantic segmentation
ImageGPT (blog post):
- (un)conditional image generation with ImageGPTForCausalLM
- linear probing with ImageGPT
LUKE (paper):
- fine-tuning LukeForEntityPairClassification on a custom relation extraction dataset using PyTorch Lightning
LayoutLM (paper):
- fine-tuning LayoutLMForTokenClassification on the FUNSD dataset
- fine-tuning LayoutLMForSequenceClassification on the RVL-CDIP dataset
- adding image embeddings to LayoutLM during fine-tuning on the FUNSD dataset
LayoutLMv2 (paper):
- fine-tuning LayoutLMv2ForSequenceClassification on RVL-CDIP
- fine-tuning LayoutLMv2ForTokenClassification on FUNSD
- fine-tuning LayoutLMv2ForTokenClassification on FUNSD using the 🤗 Trainer
- performing inference with LayoutLMv2ForTokenClassification on FUNSD
- true inference with LayoutLMv2ForTokenClassification (when no labels are available) + Gradio demo
- fine-tuning LayoutLMv2ForTokenClassification on CORD
- fine-tuning LayoutLMv2ForQuestionAnswering on DOCVQA
LayoutLMv3 (paper):
- fine-tuning LayoutLMv3ForTokenClassification on the FUNSD dataset
LayoutXLM (paper):
- fine-tuning LayoutXLM on the XFUND benchmark for token classification
- fine-tuning LayoutXLM on the XFUND benchmark for relation extraction
MarkupLM (paper):
- inference with MarkupLM to perform question answering on web pages
- fine-tuning MarkupLMForTokenClassification on a toy dataset for NER on web pages
Mask2Former (paper):
- performing inference with Mask2Former for universal image segmentation:
MaskFormer (paper):
- performing inference with MaskFormer (both semantic and panoptic segmentation):
- fine-tuning MaskFormer on a custom dataset for semantic segmentation
OneFormer (paper):
- performing inference with OneFormer for universal image segmentation:
Perceiver IO (paper):
- showcasing masked language modeling and image classification with the Perceiver
- fine-tuning the Perceiver for image classification
- fine-tuning the Perceiver for text classification
- predicting optical flow between a pair of images with PerceiverForOpticalFlow
- auto-encoding a video (images, audio, labels) with PerceiverForMultimodalAutoencoding [![Open In Colab](https://colab.rese

11.8K ·GitHub

Transformers-Tutorials

这是什么？

这是一组 Jupyter 笔记本演示，展示如何在 PyTorch 中使用 HuggingFace 的 Transformers 库，涵盖 BERT、GPT-2、LayoutLM、DETR、Donut 等众多模型。

这是一个库，还是一组教程？

这是一组教程，不是需要安装的库：每个模型都有一个或多个可以直接运行的 Jupyter 笔记本，用来演示推理或微调。

我该如何运行其中的某个笔记本？

打开你感兴趣的模型对应的笔记本（大多数都带有 Open In Colab 徽章），直接在 Google Colab 中运行；或者克隆仓库，用 Jupyter 在本地运行。

运行这些笔记本需要先安装什么？

需要 Python 和 PyTorch，以及 HuggingFace Transformers 库（某些特定演示还可能需要 PyTorch Lightning 或 Gradio 等额外组件）。

这些笔记本开箱即用，对新手友好吗？

如果你刚接触 HuggingFace/Transformers，作者建议先学习他们提供的免费课程（涵盖 BERT、GPT-2、T5 和 BART 等架构），再来看这些演示。

这组教程面向哪些人？

面向已经了解一些深度学习、希望通过实际示例学习如何微调或运行特定 Transformer 模型推理的开发者和研究人员。

涵盖了多少个模型？

数十个，涉及自然语言处理（BERT、GPT-J、LUKE）、视觉（ViT、DETR、DINO、MaskFormer）、文档 AI（LayoutLM、Donut、MarkupLM）、音频（Audio Spectrogram Transformer）和多模态（GIT、Perceiver IO）等领域。

我能用这些笔记本在自己的数据集上微调模型吗？

可以。许多笔记本（例如在 FUNSD 上微调 LayoutLM、在自定义气球数据集上微调 DETR、用 Donut 做文档解析）都是按微调示例组织的，你可以改成适用于自己数据的版本。

有没有涵盖文档理解模型的演示？

有。文档 AI 模型的内容很丰富，包括 LayoutLM、LayoutLMv2/v3、LayoutXLM、Donut 和 MarkupLM，可用于表单理解、文档分类和视觉问答等任务。

如果我刚接触 Transformers，应该从哪里开始？

先学习 README 中提到的 HuggingFace 免费课程，然后选一个 BERT 或 ViT 等知名模型的笔记本，再去看更专业的架构。

What are similar projects?

dinov3, InternVL, DeepLearning, DeepLearningExamples, unilm, deep-learning-for-image-processing, learnopencv, dinov2

Which topics is it in?

gpt-2, bert, vision-transformer, image-classification, object-detection, transformers, ocr, embeddings, gpt, document-processing, jupyter notebook, fine-tuning

Creator: NielsRogge
Size: 264.3 MB
Stars: 11.8K
Updated: CommitSep 17
Age: 6y
Commits · 90 Days

Transformers-Tutorials README

Transformers-Tutorials

Transformers-Tutorials (NielsRogge/Transformers-Tutorials)