25.5K ·GitHub
What is this?
OmniParser parses GUI screenshots into structured, labeled elements (icons, text, interactable regions) so a vision model like GPT-4V can accurately decide what to click or type on a screen.
OmniParser parses GUI screenshots into structured, labeled elements (icons, text, interactable regions) so a vision model like GPT-4V can accurately decide what to click or type on a screen.Is this a ready-made agent app, or a toolkit I plug into my own agent?It is a perception toolkit (a YOLOv9-based region detector plus an icon captioning model) that you call from a notebook, a Gradio demo, or 'OmniTool' which wires it up to a vision model to drive a VM.
It is a perception toolkit (a YOLOv9-based region detector plus an icon captioning model) that you call from a notebook, a Gradio demo, or 'OmniTool' which wires it up to a vision model to drive a VM.How do I get it running?Clone the repo, install the environment, then download the detector and caption weights into the weights folder; a Gradio demo and demo.ipynb are included as examples.
Clone the repo, install the environment, then download the detector and caption weights into the weights folder; a Gradio demo and demo.ipynb are included as examples.Which vision models can I pair it with via OmniTool?OmniTool works out of the box with OpenAI (4o/o1/o3-mini), DeepSeek (R1), Qwen (2.5VL), or Anthropic Computer Use.
OmniTool works out of the box with OpenAI (4o/o1/o3-mini), DeepSeek (R1), Qwen (2.5VL), or Anthropic Computer Use.Any licensing gotchas with the model weights?Yes: the newer icondetectv3 detector and caption models are MIT, but earlier Ultralytics-based icon detectors keep their original AGPL license, so check which weight version you pull.
Yes: the newer icondetectv3 detector and caption models are MIT, but earlier Ultralytics-based icon detectors keep their original AGPL license, so check which weight version you pull.What is new in OmniParser V2?V2 adds finer small-icon detection and predicts whether each UI element is interactable, and it set a new state of the art of 39.5% on the Screen Spot Pro grounding benchmark.
V2 adds finer small-icon detection and predicts whether each UI element is interactable, and it set a new state of the art of 39.5% on the Screen Spot Pro grounding benchmark.What is OmniTool?OmniTool is a companion app that lets you control a Windows 11 VM using OmniParser's screen parsing combined with a vision model of your choice, such as GPT-4o or Claude Computer Use.
OmniTool is a companion app that lets you control a Windows 11 VM using OmniParser's screen parsing combined with a vision model of your choice, such as GPT-4o or Claude Computer Use.What benchmark results back this up?OmniParser topped the Windows Agent Arena leaderboard and achieved 39.5% on the newer Screen Spot Pro grounding benchmark with V2.
OmniParser topped the Windows Agent Arena leaderboard and achieved 39.5% on the newer Screen Spot Pro grounding benchmark with V2.What are similar projects?self-operating-computer, Qwen-Drive-1.0, UI-TARS, LLaVA, ZDTaichu5.0-9B, fara, Qwen3-VL, Agent-S
self-operating-computer, Qwen-Drive-1.0, UI-TARS, LLaVA, ZDTaichu5.0-9B, fara, Qwen3-VL, Agent-SWhich topics is it in?gui-agent, vision, gpt-4, vision-language-model, gradio, computer-use, object-detection, microsoft, qwen, jupyter notebook, multimodal, deepseek
gui-agent, vision, gpt-4, vision-language-model, gradio, computer-use, object-detection, microsoft, qwen, jupyter notebook, multimodal, deepseek