14.6K ·GitHub
What is this?
AR Cut & Paste is a research prototype that lets you point your phone's camera at a real-world object, cut it out using AR and machine learning, and paste it directly into Photoshop as a new layer.
AR Cut & Paste is a research prototype that lets you point your phone's camera at a real-world object, cut it out using AR and machine learning, and paste it directly into Photoshop as a new layer.Is this one app or several pieces I need to run together?It's three independent modules working together: a mobile AR app, a local bridge server, and a separate object-detection/background-removal service.
It's three independent modules working together: a mobile AR app, a local bridge server, and a separate object-detection/background-removal service.How do I get the whole pipeline running?Enable Remote Connection in Photoshop's plug-in preferences, set up the salience detection service (self-hosted or community endpoint), then follow the setup instructions in the /server folder and the /app folder.
Enable Remote Connection in Photoshop's plug-in preferences, set up the salience detection service (self-hosted or community endpoint), then follow the setup instructions in the /server folder and the /app folder.Do I need special hardware to run this myself?To self-host the BASNet salience detection model you need a machine with a CUDA GPU; otherwise you can skip that and use a public community-hosted endpoint instead.
To self-host the BASNet salience detection model you need a machine with a CUDA GPU; otherwise you can skip that and use a public community-hosted endpoint instead.Is this ready to use as a everyday Photoshop tool?No, the author is explicit that this is a research prototype, not a polished consumer tool, and results depend on the Photoshop document having enough background detail for SIFT feature matching to work correctly.
No, the author is explicit that this is a research prototype, not a polished consumer tool, and results depend on the Photoshop document having enough background detail for SIFT feature matching to work correctly.Can I run the background-removal service without owning a GPU?Yes — running your own BASNet model service requires a CUDA GPU, but if you don't have one you can point the local server at the public community-hosted endpoint instead.
Yes — running your own BASNet model service requires a CUDA GPU, but if you don't have one you can point the local server at the public community-hosted endpoint instead.What does BASNet do in this pipeline?BASNet (CVPR 2019) is the deep learning model used for salience detection and background removal, which isolates the object the camera is pointed at so it can be cut out.
BASNet (CVPR 2019) is the deep learning model used for salience detection and background removal, which isolates the object the camera is pointed at so it can be cut out.How does the local server know where to paste the object in Photoshop?It uses SIFT-based feature matching (via screenpoint) to figure out where the camera is pointing relative to the open Photoshop document, then pastes the cut-out object as a new layer there.
It uses SIFT-based feature matching (via screenpoint) to figure out where the camera is pointing relative to the open Photoshop document, then pastes the cut-out object as a new layer there.Is there a simpler, non-developer version of this idea?The README notes that a separate, easier-to-use app based on this research exists and points readers to it for a less technical setup.
The README notes that a separate, easier-to-use app based on this research exists and points readers to it for a less technical setup.What are similar projects?VisionHOPE, Detectron, X-AnyLabeling, detectron2, Grounded-Segment-Anything, opencv, albumentations, TaskMatrix
VisionHOPE, Detectron, X-AnyLabeling, detectron2, Grounded-Segment-Anything, opencv, albumentations, TaskMatrixWhich topics is it in?prototype, augmented-reality, image-segmentation, ml, segmentation, image-editing, object-detection, gpu-acceleration, image-processing, mobile-app, computer-vision, cuda
prototype, augmented-reality, image-segmentation, ml, segmentation, image-editing, object-detection, gpu-acceleration, image-processing, mobile-app, computer-vision, cuda