RelateAnything README

The project's README from github.com/Maelic/RelateAnything. As one markdown file: README.md.

RelateAnything

Real-time open-vocabulary relation prediction from any inputs.

arXiv models demo ci license

Paper · Project page · Models · Training data · Documentation

Give RelateAnything an image, object regions, and the relations you want to look for. It returns scored (subject, relation, object) triplets, such as person → riding → horse.

https://github.com/user-attachments/assets/76544f31-53ac-407b-9445-157e83946938

The same relation model across five scenes, with tracked regions and relations. Clip credits · Reproduce the video

Try the browser demo — no installation; inference runs on your device using WebGPU or WASM.

Quickstart · Models · Deployment · Performance · Results · How it works

Quickstart

Use Python 3.12+. Install a CUDA-enabled PyTorch build for NVIDIA GPU inference, or use device="cpu" in the example below.

git clone https://github.com/Maelic/RelateAnything.git
cd RelateAnything
pip install -e ".[hub]"

Run this from the repository root. The sample photo is included; the two example boxes identify the person and the horse.

import numpy as np
from PIL import Image
from relsgg import RelateAnything

model = RelateAnything.from_pretrained("maelic/relsgg-vits16plus", device="cuda")
image = Image.open("assets/reel/images/horse.jpg").convert("RGB")
boxes = np.array([[470, 130, 650, 630], [90, 310, 1010, 875]], dtype=np.float32)

# Boxes are [x1, y1, x2, y2] in original-image pixels.
# Labels are optional and only make the printed triplets easier to read.
for triplet in model.predict(image, boxes, box_labels=["person", "horse"], topk=5):
    print(triplet)

# Change what relations to look for, without retraining.
model.set_vocabulary(["riding", "carrying a rider", "beside", "in front of"])
graphs = model.predict(image, boxes, box_labels=["person", "horse"], decompose=True)
print(graphs["spatial"])
print(graphs["semantic"])

Released checkpoints include the backbone configuration and text encoder; no gated DINOv3 login is needed for inference. Replace the sample image and boxes with your own inputs; the API also supports masks and a larger built-in relation vocabulary.

API guide: inputs, vocabularies, masks, and scores · Installation and offline use

Models

Start with relsgg-vits16plus, the recommended balance of quality and speed. All three checkpoints use the same training recipe; their vision backbones differ.

Checkpoint Backbone Parameters
relsgg-vits16 DINOv3 ViT-S/16 46.1 M
relsgg-vits16plus DINOv3 ViT-S/16+ 53.2 M
relsgg-vitb16 DINOv3 ViT-B/16 113.8 M

Compare model quality and speed. Each model card includes evaluation results and calibration details.

Deployment

After downloading the relation graph and building the local detector as explained in the deployment guide:

# CPU image demo; omit --image to use a webcam.
python deploy/demo_webcam.py --dist deploy/dist/relsgg-vits16plus --image photo.jpg

# NVIDIA GPU, after building both TensorRT engines.
python deploy/demo_webcam.py --dist deploy/dist/relsgg-vits16plus \
  --backend tensorrt --device cuda --image photo.jpg --decompose

The relation model consumes regions; it does not detect objects itself. Demo detectors are rebuilt locally and have separate licenses. The exported ONNX/TensorRT graph scores boxes; native mask inputs are supported by the PyTorch API. Deployment contracts and limitations.

Performance

On an RTX 3080 Laptop GPU, the recommended ViT-S+ model takes 17.2 ms per image with TensorRT FP32, including preprocessing, transfers and triplet decoding. Object regions are provided as input.

With YOLO26m detection included, the full pipeline takes 36.5 ms per image. Detection uses PyTorch FP32; ViT-S+ relations use TensorRT FP32.

Warm median latency, batch 1, 35 relation types, 6 sample images; loading and setup excluded.

All models and runtimes · YOLO26, YOLO-World and YOLOE comparison · Paper pipeline benchmarks

Results

The paper evaluates relation prediction across datasets, open vocabularies and spatial reasoning tasks. Selected transfer results for ViT-S+, using ground-truth boxes and one relation per ordered pair:

Test set OvSGTR F1@50 RelateAnything F1@50
VG150 16.5 36.9
PSG 13.5 34.7
IndoorVG 20.2 37.8

Both models use the same evaluator and vocabulary. OvSGTR receives object labels; RelateAnything does not. The full results cover stronger baselines including ROBIN-3B, zero-shot controls and the complete evaluation.

Evaluation protocol · Scoring guide

How it works

Figure 2: RelateAnything's visual path, text embedding bank, predicate-conditioned scoring gate, and training objective

Figure 2 from the paper. Click to enlarge.

A DINOv3 backbone reads the image once. Regions become visual tokens; a pair sampler selects candidate relations; a transformer refines them using the scene. A vocabulary head scores each pair against embeddings of your predicate strings, mixing spatial and semantic features. Swapping that embedding bank changes the vocabulary without retraining the vision model.

Architecture guide

Data and evaluation packs

Reproduce and contribute

The documentation index covers the API, architecture, training, data and deployment. Read evaluation pitfalls before comparing results or changing the scoring path.

License

Full third-party notices and credits

Citation

@article{neau2026relateanything,
  title   = {RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs},
  author  = {Neau, Ma\"elic},
  journal = {arXiv preprint arXiv:2609.12552},
  eprint  = {2609.12552},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url     = {https://arxiv.org/abs/2609.12552},
  year    = {2026}
}

RelateAnything (Maelic/RelateAnything)

deployment
Searching Database…

1.1K ·GitHub
RelateAnything
What is this?RelateAnything is a real-time, open-vocabulary relation prediction model: give it an image, object regions and the relations to look for, and it returns scored (subject, relation, object) triplets like person → riding → horse.
So what am I actually using — a Python library, a web app, a service?It is the code release behind a research paper: a PyTorch API with pretrained checkpoints, plus training and evaluation tooling, an ONNX graph for TensorRT and OpenVINO deployment, and a no-install browser demo.
How do I run it on my own photo?Run the quickstart from the repository root: a sample photo and two example boxes (the person and the horse) are included; swap in your own image and boxes. Masks and a larger built-in relation vocabulary are supported too.
Do I need anything special installed before running it?Python 3.12+ and PyTorch — install the CUDA-enabled build for NVIDIA GPU inference, or run on CPU with device='cpu'. Released checkpoints bundle the backbone and text encoder, so no gated DINOv3 login is needed for inference.
What's the catch — can it find objects in a photo on its own?No — it does no object detection: you supply regions from a detector, a segmenter, or by hand, and masks are optional. Demo detectors are rebuilt locally and have separate licenses.
If I try the browser demo, do my images get sent to a server?No — the demo runs inference entirely on your device using WebGPU or WASM, so images never leave it.
Is the code and the model free to use?Free to use with a catch: the code is Apache-2.0, but the weights are Meta DINOv3 derivatives, the RA-4M annotations carry Gemma Terms of Use, and the demo detectors are AGPL-3.0 and excluded from release artifacts.
Do I need an NVIDIA GPU for this?No. CPU inference works out of the box, laptops can run local ONNX or OpenVINO demos with an image or a webcam, and the browser demo needs nothing installed — the NVIDIA GPU with TensorRT route is simply the fastest.
What does open-vocabulary mean here — can I search for my own relations?Yes — you supply the relation phrases at inference time, with no retraining. A small text encoder embeds them once and no language model runs per frame; a larger built-in relation vocabulary is also available.
How fast does it actually run?On an RTX 3080 Laptop GPU the recommended ViT-S+ model takes 17.2 ms per image with TensorRT FP32, including preprocessing; with YOLO26m detection added, the full pipeline is 36.5 ms per image.
There are three checkpoints — which one should I pick?Start with relsgg-vits16plus (53.2M parameters), the recommended balance of quality and speed. relsgg-vits16 (46.1M) and relsgg-vitb16 (113.8M) share the same training recipe and differ only in their DINOv3 ViT backbones.
How does it compare to earlier open-vocabulary relation models?By a wide margin: F1@50 of 36.9 versus OvSGTR's 16.5 on VG150, 34.7 vs 13.5 on PSG and 37.8 vs 20.2 on IndoorVG, using the same evaluator — and unlike OvSGTR, RelateAnything receives no object labels.
What are similar projects?YOLOX, High-Angle_Robust_Fast_FaceAlignment, yolov3, SolarWM, NanoJev, pointdit, yolov5, segment-anything
Which topics is it in?tensorrt, pretrained-models, onnx, training, deployment, dataset, evaluation, transformer, real-time, inference, computer-vision, cuda
Creator
Maelic
Size
27.5 MB
Stars
1.1K
Updated
CommitOct 12
Age
1mo
Commits · 90 Days