What is this?
Microsoft's research hub for large-scale self-supervised pre-training: a family of foundation models spanning NLP, vision, speech, and multimodal tasks across 100+ languages - including BEiT, LayoutLM, Kosmos, MiniLM, TrOCR, and WavLM.
Microsoft's research hub for large-scale self-supervised pre-training: a family of foundation models spanning NLP, vision, speech, and multimodal tasks across 100+ languages - including BEiT, LayoutLM, Kosmos, MiniLM, TrOCR, and WavLM.Is this one library I install, or a collection of separate models?It is an umbrella repo rather than a single installable package: each model (BEiT, LayoutLM, Kosmos, TrOCR and so on) is its own project with its own code and README, and many are also published on HuggingFace.
It is an umbrella repo rather than a single installable package: each model (BEiT, LayoutLM, Kosmos, TrOCR and so on) is its own project with its own code and README, and many are also published on HuggingFace.How do I get started with one of these models?There is no single setup for the whole hub - open the README of the model you want and follow its instructions. Many models are also on HuggingFace, so you can often load a pretrained checkpoint directly from there.
There is no single setup for the whole hub - open the README of the model you want and follow its instructions. Many models are also on HuggingFace, so you can often load a pretrained checkpoint directly from there.What do I need on my machine to run these?The models are Python research code and each sub-project ships its own requirements; portions of the source are built on the HuggingFace transformers stack, so a working Python environment is the baseline.
The models are Python research code and each sub-project ships its own requirements; portions of the source are built on the HuggingFace transformers stack, so a working Python environment is the baseline.What is the catch - is there one entry point for everything?No: the root README is mostly an index and news feed, and models live in separate sub-repos with their own dependencies and setup, so expect per-model docs instead of one unified API.
No: the root README is mostly an index and news feed, and models live in separate sub-repos with their own dependencies and setup, so expect per-model docs instead of one unified API.What is TrOCR?TrOCR is a transformer-based OCR approach built on pre-trained BEiT and RoBERTa models. BASE and LARGE releases target scene text recognition, and small models for handwritten and printed text come with a 3x inference speedup; TrOCR is also available on HuggingFace.
TrOCR is a transformer-based OCR approach built on pre-trained BEiT and RoBERTa models. BASE and LARGE releases target scene text recognition, and small models for handwritten and printed text come with a 3x inference speedup; TrOCR is also available on HuggingFace.What is Kosmos?A family of multimodal LLMs: Kosmos-1 perceives general modalities, learns in context (few-shot) and follows instructions (zero-shot); Kosmos-2 grounds the MLLM to the world; Kosmos-2.5 is a multimodal literate model for machine reading of text-intensive images.
A family of multimodal LLMs: Kosmos-1 perceives general modalities, learns in context (few-shot) and follows instructions (zero-shot); Kosmos-2 grounds the MLLM to the world; Kosmos-2.5 is a multimodal literate model for machine reading of text-intensive images.What is BEiT-3?A general-purpose multimodal foundation model and a major milestone of the Big Convergence across tasks, languages and modalities; per the README it achieves state-of-the-art transfer performance on both vision and vision-language tasks. Pretrained models and code were released in March 2023.
A general-purpose multimodal foundation model and a major milestone of the Big Convergence across tasks, languages and modalities; per the README it achieves state-of-the-art transfer performance on both vision and vision-language tasks. Pretrained models and code were released in March 2023.Is this repo still updated?Yes - the latest news entry is December 2024 (RedStone, curated training data for LLMs), and releases have been continuous since 2019, including Kosmos-2.5, TextDiffuser-2 and LongNet in 2023.
Yes - the latest news entry is December 2024 (RedStone, curated training data for LLMs), and releases have been continuous since 2019, including Kosmos-2.5, TextDiffuser-2 and LongNet in 2023.What are similar projects?Transformers-Tutorials, dinov3, LongCat-Next, seamless_communication, datasets, DeepSeek-OCR, StartLux-Decision, transformers
Transformers-Tutorials, dinov3, LongCat-Next, seamless_communication, datasets, DeepSeek-OCR, StartLux-Decision, transformersWhich topics is it in?self-supervised-learning, foundation-models, natural-language-processing, text-to-speech, ocr, embeddings, multimodal, nlp, artificial-intelligence, huggingface, computer-vision, pytorch
self-supervised-learning, foundation-models, natural-language-processing, text-to-speech, ocr, embeddings, multimodal, nlp, artificial-intelligence, huggingface, computer-vision, pytorch