Detect graphic user interface (GUI) elements from the GUI image and recognize the repetitive layout structure through grouping the elemetns.
On GUIs, we do not just see a collection of separated texts, images, buttons, etc. Instead,we see perceptual groups of GUI widgets, such as card, list, tab and menu.
Although humans can intuitively see perceptual groups of GUI widgets, current computational approaches are rather limited in partitionnig a GUI into meaningful groups of widget elements. Thus, the GUI Perceptual Grouping is proposed to mimic how human beings perceive the GUI to recognize and gather the atomic GUI elements into structured groups.
The approch's design uses a well-established psychological theory, the Gestalt principles of perception, for reference to develop its core algorithms.
Gestalt theory systematically explains how humans see the whole rather than individual and unrelated parts.
It includes a set of principles of grouping, among which connectedness, similarity, proximity and continuity are the most essential ones.
Connectedness - We perceive elements connected by uniform visual properties as being more related than those not connected
Similarity - Elements are perceptually grouped together if they are similar to each other
Proximity - When people see an assortment of objects, they tend to perceive objects that are close (proximate) to each other as a group
Contiuity - Elements arranged in a line or curve are perceived to be more related
Gestalt principles also greatly influence UI design, but they may be represented in a disctinct way.
Connectedness - Box container that contains multiple widgets within it, and all the enclosed widgets are perceived as in the same group
Similarity - Similarity can be observed in aspects of various visual cues, such as size, color, shape or position
Proximity - Some groups are close to each other and similar in terms of the number and layout of the contained widgets
Contiuity - Some detection errors are likely to be spotted if a GUI area or a widget aligns with all the widgets in a perceptual group in a line but is not gathered into that group
The GUI Perceptual Grouping is based on the UIED to detect the GUI element, but both UIED and the grouping approach are unsupervised that requires NO traning process and massive data preparation to deploy extremly easily.
- Python
- Pandas
- sklearn
- OpenCV
Note that no strict version required for the dependency as long as the program running well in your machine.
The approach is purely based on computer vision techniques, it thus only needs the input of a GUI image and then outputs the GUI elements and their grouping.
The basic usage is:
gui = GUI(img_file="data/input/2.jpg", output_dir="data/output")
gui.detect_element(is_ocr=True, is_non_text=True, is_merge=True)
gui.visualize_element_detection()
gui.recognize_layout()
gui.visualize_layout_recognition()
