Transformers-Tutorials README

Transformers-Tutorials

Hi there!

This repository contains demos I made with the Transformers library by 🤗 HuggingFace. Currently, all of them are implemented in PyTorch.

NOTE: if you are not familiar with HuggingFace and/or Transformers, I highly recommend to check out our free course, which introduces you to several Transformer architectures (such as BERT, GPT-2, T5, BART, etc.), as well as an overview of the HuggingFace libraries, including Transformers, Tokenizers, Datasets, Accelerate and the hub.

For an overview of the ecosystem of HuggingFace for computer vision (June 2022), refer to this notebook with corresponding video.

Currently, it contains the following demos:

Audio Spectrogram Transformer (paper):
- performing inference with ASTForAudioClassification to classify audio.
BERT (paper):
- fine-tuning BertForTokenClassification on a named entity recognition (NER) dataset.
- fine-tuning BertForSequenceClassification for multi-label text classification.
BEiT (paper):
- understanding BeitForMaskedImageModeling
CANINE (paper):
- fine-tuning CanineForSequenceClassification on IMDb
CLIPSeg (paper):
- performing zero-shot image segmentation with CLIPSeg
Conditional DETR (paper):
- performing inference with ConditionalDetrForObjectDetection
- fine-tuning ConditionalDetrForObjectDetection on a custom dataset (balloon)
ConvNeXT (paper):
- fine-tuning (and performing inference with) ConvNextForImageClassification
DINO (paper):
- visualize self-attention of Vision Transformers trained using the DINO method
DETR (paper):
- performing inference with DetrForObjectDetection
- fine-tuning DetrForObjectDetection on a custom object detection dataset
- evaluating DetrForObjectDetection on the COCO detection 2017 validation set
- performing inference with DetrForSegmentation
- fine-tuning DetrForSegmentation on COCO panoptic 2017
DPT (paper):
- performing inference with DPT for monocular depth estimation
- performing inference with DPT for semantic segmentation
Deformable DETR (paper):
- performing inference with DeformableDetrForObjectDetection
DiT (paper):
- performing inference with DiT for document image classification
Donut (paper):
- performing inference with Donut for document image classification
- fine-tuning Donut for document image classification
- performing inference with Donut for document visual question answering (DocVQA)
- performing inference with Donut for document parsing
- fine-tuning Donut for document parsing with PyTorch Lightning
GIT (paper):
- performing inference with GIT for image/video captioning and image/video question-answering
- fine-tuning GIT on a custom image captioning dataset
GLPN (paper):
- performing inference with GLPNForDepthEstimation to illustrate monocular depth estimation
GPT-J-6B (repository):
- performing inference with GPTJForCausalLM to illustrate few-shot learning and code generation
GroupViT (repository):
- performing inference with GroupViTModel to illustrate zero-shot semantic segmentation
ImageGPT (blog post):
- (un)conditional image generation with ImageGPTForCausalLM
- linear probing with ImageGPT
LUKE (paper):
- fine-tuning LukeForEntityPairClassification on a custom relation extraction dataset using PyTorch Lightning
LayoutLM (paper):
- fine-tuning LayoutLMForTokenClassification on the FUNSD dataset
- fine-tuning LayoutLMForSequenceClassification on the RVL-CDIP dataset
- adding image embeddings to LayoutLM during fine-tuning on the FUNSD dataset
LayoutLMv2 (paper):
- fine-tuning LayoutLMv2ForSequenceClassification on RVL-CDIP
- fine-tuning LayoutLMv2ForTokenClassification on FUNSD
- fine-tuning LayoutLMv2ForTokenClassification on FUNSD using the 🤗 Trainer
- performing inference with LayoutLMv2ForTokenClassification on FUNSD
- true inference with LayoutLMv2ForTokenClassification (when no labels are available) + Gradio demo
- fine-tuning LayoutLMv2ForTokenClassification on CORD
- fine-tuning LayoutLMv2ForQuestionAnswering on DOCVQA
LayoutLMv3 (paper):
- fine-tuning LayoutLMv3ForTokenClassification on the FUNSD dataset
LayoutXLM (paper):
- fine-tuning LayoutXLM on the XFUND benchmark for token classification
- fine-tuning LayoutXLM on the XFUND benchmark for relation extraction
MarkupLM (paper):
- inference with MarkupLM to perform question answering on web pages
- fine-tuning MarkupLMForTokenClassification on a toy dataset for NER on web pages
Mask2Former (paper):
- performing inference with Mask2Former for universal image segmentation:
MaskFormer (paper):
- performing inference with MaskFormer (both semantic and panoptic segmentation):
- fine-tuning MaskFormer on a custom dataset for semantic segmentation
OneFormer (paper):
- performing inference with OneFormer for universal image segmentation:
Perceiver IO (paper):
- showcasing masked language modeling and image classification with the Perceiver
- fine-tuning the Perceiver for image classification
- fine-tuning the Perceiver for text classification
- predicting optical flow between a pair of images with PerceiverForOpticalFlow
- auto-encoding a video (images, audio, labels) with PerceiverForMultimodalAutoencoding [![Open In Colab](https://colab.rese

11.8K ·GitHub

Transformers-Tutorials

What is this?

A collection of Jupyter notebook demos showing how to use HuggingFace's Transformers library in PyTorch, covering models like BERT, GPT-2, LayoutLM, DETR, Donut and many more.

Is this a library or a set of tutorials?

It's a tutorial collection, not a library you install: each model gets one or more runnable Jupyter notebooks demonstrating inference or fine-tuning.

How do I run one of these notebooks?

Open the notebook for the model you're interested in (most have an Open In Colab badge) and run it directly in Google Colab, or clone the repo and run it locally with Jupyter.

What do I need installed to run these?

Python and PyTorch, plus the HuggingFace Transformers library (and sometimes extras like PyTorch Lightning or Gradio for specific demos).

Are these notebooks beginner-friendly out of the box?

If you're new to HuggingFace/Transformers, the author recommends first taking their free course covering architectures like BERT, GPT-2, T5 and BART before diving into these demos.

Who is this collection for?

Developers and researchers who already know some deep learning and want hands-on examples of fine-tuning or running inference with specific Transformer models.

How many models are covered?

Dozens, spanning NLP (BERT, GPT-J, LUKE), vision (ViT, DETR, DINO, MaskFormer), document AI (LayoutLM, Donut, MarkupLM), audio (Audio Spectrogram Transformer) and multimodal (GIT, Perceiver IO).

Can I fine-tune a model on my own dataset using these notebooks?

Yes, many notebooks (e.g. LayoutLM on FUNSD, DETR on a custom balloon dataset, Donut for document parsing) are structured as fine-tuning examples you can adapt to your own data.

Is there a demo covering document understanding models?

Yes, there's extensive coverage of document AI models like LayoutLM, LayoutLMv2/v3, LayoutXLM, Donut and MarkupLM for tasks like form understanding, document classification and visual question answering.

Where should I start if I'm new to Transformers?

Start with the HuggingFace free course mentioned in the README, then pick a notebook for a well-known model like BERT or ViT before moving to more specialized architectures.

What are similar projects?

dinov3, InternVL, DeepLearning, DeepLearningExamples, unilm, deep-learning-for-image-processing, learnopencv, dinov2

Which topics is it in?

gpt-2, bert, vision-transformer, image-classification, object-detection, transformers, ocr, embeddings, gpt, document-processing, jupyter notebook, fine-tuning

Creator: NielsRogge
Size: 264.3 MB
Stars: 11.8K
Updated: CommitSep 17
Age: 6y
Commits · 90 Days

Transformers-Tutorials README

Transformers-Tutorials

Transformers-Tutorials (NielsRogge/Transformers-Tutorials)