Learning Structural Similarity of User Interface Layouts using Graph Networks

Datasets

RICO Dataset

Download RICO dataset from rico (Optional)
We use semantic UI screenshots and annotations. Simplified annotation for semantic RICO UIs is given in data/rico_box_info_list.pkl
Data partition sets (train/gallery/query) used for all experiments are in data/UI_data.p and data/UI_test_data.p

GoogleUI Dataset

We release GoogleUI dataset. GoogleUI is a new dataset of 18.5K UX designs collected from web.
Dataset/annotations/descriptions can be obtained from Google Drive

Evaluation and trained model

To evaluate the model:

mkdir trained_models
Download trained models from here into trained_models/
Prior to evaluation/training, prepare graph represenatations for UIs following steps below:
- Run python graph_scripts/cal_geometry_feat.py. This will compute the geometric features for all rico UIs
- Run python graph_scripts/build_geomerty_graph.py. This will pre-construct the graph data for UIs; saved under graph_data/
Run python evaluate.py to get the performance metrics: top-k mIoU and mPixAcc

Training GCN-CNN

To train GCN-CNN model

python train.py --batch_size 10 --decoder_model 'strided' --dim 1024 \
--use_directed_graph True

For faster dataloading and training, it is recommended to pre-compute the 25-Channel representations for all RICO UIs
- To do so: run python compute_25Chan_Imgs.py
- This will save all 25 Channel represenations for all UIs into data/25ChanImages
To train GCN-CNN model using pre-computed 25-Channel representations

python train.py --batch_size 10 --decoder_model 'strided' --dim 1024 \
--use_directed_graph True \
--use_precomputed_25Chan_imgs True\
--Channel25_img_dir 'data\25ChanImages'

The model is saved into outpur_dir. It performs retrieval evaluation every N epoch and logs results into output_dir\results.txt

Fine-tune/train with triplet supervision (GCN-CNN-TRI)

To train GCN-CNN-TRI model, we need to generate triplets for training which we provide here. Download it under Triplets/
If you want to generate your own triplets, See Triplets
Download the pretrained GCN_CNN model to fine-tune from here into trained_models/.

python train_TRI.py --batch_size 10 --decoder_model 'strided' --dim 1024 \
--use_directed_graph True \
--use_precomputed_25Chan_imgs True\
--Channel25_img_dir 'data\25ChanImages' \
--apn_dict_path 'Triplets/apn_dict_48K_pthres60.pkl'

The model is saved into outpur_dir. It performs retrieval evaluation every N epoch and logs results into output_dir\result.txt

Reference

@inproceedings{gcncnn_eccv2020,
 title={Learning Structural Similarity of User Interface Layouts using Graph Networks},
 author={Dipu Manandhar, Dan Ruta, and John Collomosse},
 booktitle={ECCV},
 year={2020}}

Acknowlegdement

This repo re-uses part of the code from ltguo19/VSUA-Captioning.

EmanuelKuhn/gcn-cnn