A library facilitating the generation of topological representations for image segments. The topological graph is created based on segments and their visual embeddings as nodes. This segment topology retains not only spatial but also semantic information, making it very useful for various tasks in visual robotics[1] and localization.
GraphVision provides graphical representations that enable us to perform visual queries and interact with our segmentation graph without the usual preprocessing hassle. GraphVision handles everything and also offers functionalities to visualize the segment topology and perform visual queries on the graph.
Go though the Colab Notebook to get a detailed overview of the library.
The library requirements are flexible but still its suggested to use a virtual environment.
pip install graphvisionRead an Image
image_bgr = cv2.imread("image.jpg")
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)Initialize the generator, call the segmentation model, generate the masks and that is it! You are ready create the topological graph for your segments.
from graphvision import Generator, SegmentGraph
gen = Generator()
mask_gen = gen.SAM()
segments = gen.generate_masks(mask_gen, image_rgb)You can visualize or plot it using gen.plot_segments(segments)
Create the topological graph
sg = SegmentGraph(segments, image_rgb)
G, centroids = sg.get_topology(dist_thres=150) # Other Optional Parameters: area_percent, add_to_bboxThe graph G can be returned as a networkx (default) or as a PyTorch geometric object. This graphical representation can now be leveraged in various vision tasks such as object localization and environment mapping in robotics, based on both spatial and semantic features..
Query the segment graph
You can also perform visual queries on the graph to locate objects dependent on other nodes (neighbouring objects). This a naive implementation, so please go through the Colab notebook to understand it in greater detail.
result = sg.query_segment_graph(G, query_pair=("Girl", "plant"), show_legend=True)