What is this?
This is the open-sourced research code for MiniGPT-4 and MiniGPT-v2, two vision-language models that use a large language model as the interface for understanding and chatting about images.
This is the open-sourced research code for MiniGPT-4 and MiniGPT-v2, two vision-language models that use a large language model as the interface for understanding and chatting about images.Is this a ready-made app or research code I have to set up myself?It is a research codebase, not a packaged app: you clone the repo, download separate LLM weights and checkpoints, edit config files, and launch a local demo yourself.
It is a research codebase, not a packaged app: you clone the repo, download separate LLM weights and checkpoints, edit config files, and launch a local demo yourself.How do I get the demo running locally?Clone the repo and set up a Python environment, download the base LLM weights (Llama2 Chat 7B or Vicuna V0), download the matching pretrained MiniGPT checkpoint, point the config files at those paths, then run the demo script for MiniGPT-v2 or MiniGPT-4.
Clone the repo and set up a Python environment, download the base LLM weights (Llama2 Chat 7B or Vicuna V0), download the matching pretrained MiniGPT checkpoint, point the config files at those paths, then run the demo script for MiniGPT-v2 or MiniGPT-4.Which LLM weights do I need and how much GPU memory does it take?MiniGPT-v2 needs Llama2 Chat 7B; MiniGPT-4 needs either Vicuna V0 (7B/13B) or Llama2, downloaded via git-lfs from HuggingFace. Loaded in 8-bit by default, it needs about 23GB GPU memory for the 13B model and 11.5GB for a 7B model.
MiniGPT-v2 needs Llama2 Chat 7B; MiniGPT-4 needs either Vicuna V0 (7B/13B) or Llama2, downloaded via git-lfs from HuggingFace. Loaded in 8-bit by default, it needs about 23GB GPU memory for the 13B model and 11.5GB for a 7B model.Why can't I just download one model and run it right away?The pretrained LLM weights (Vicuna/Llama2) and the MiniGPT checkpoints are distributed separately and must be downloaded and wired together manually through config files before the demo works.
The pretrained LLM weights (Vicuna/Llama2) and the MiniGPT checkpoints are distributed separately and must be downloaded and wired together manually through config files before the demo works.What's the difference between MiniGPT-4 and MiniGPT-v2?MiniGPT-v2 is the newer, unified vision-language interface built on Llama2 Chat 7B for multi-task learning, while MiniGPT-4 (the earlier model) runs on Vicuna or Llama2 and focuses on enhanced vision-language understanding; both come from the same Vision-CAIR team.
MiniGPT-v2 is the newer, unified vision-language interface built on Llama2 Chat 7B for multi-task learning, while MiniGPT-4 (the earlier model) runs on Vicuna or Llama2 and focuses on enhanced vision-language understanding; both come from the same Vision-CAIR team.What architecture does it build on?MiniGPT-4's architecture follows BLIP-2, and the whole codebase is built on top of the Lavis library, pairing a vision encoder with the Vicuna/Llama2 language model.
MiniGPT-4's architecture follows BLIP-2, and the whole codebase is built on top of the Lavis library, pairing a vision encoder with the Vicuna/Llama2 language model.Can I try it without installing anything?Yes, both models have online demos you can click on the project pages to chat about your own images without any setup.
Yes, both models have online demos you can click on the project pages to chat about your own images without any setup.Can I lower the GPU memory requirement further?Models load in 8-bit by default (about 11.5GB for 7B, 23GB for 13B); if you have a more powerful GPU you can switch to 16-bit loading in the config for better quality.
Models load in 8-bit by default (about 11.5GB for 7B, 23GB for 13B); if you have a more powerful GPU you can switch to 16-bit loading in the config for better quality.What are similar projects?LAVIS, OraRL, Janus, Valen, ZDTaichu5.0-9B, LongCat-Next, UniVR, LLaDA-Image
LAVIS, OraRL, Janus, Valen, ZDTaichu5.0-9B, LongCat-Next, UniVR, LLaDA-ImageWhich topics is it in?vision-language, demo, model, vlm, vision-language-model, multimodal, academic-research, large-language-models, artificial-intelligence, chatbot, bsd, generative-ai
vision-language, demo, model, vlm, vision-language-model, multimodal, academic-research, large-language-models, artificial-intelligence, chatbot, bsd, generative-ai