MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities

MetaMorph is an AI tool to detect the constituent UI elements of low fidelity prototype sketches.

Dataset

MetaMorph uses

Setup and usage

MetaMorph API uses Python 3.6 and Tensorflow Object Detection API (TFOD).

To install and use MetaMorph API, follow the steps below

Download the following files to the models/ directory
- frozen_inference_graph.pb
- labels.json
Install poetry
```
pip install poetry
```
Upgrade pip, as older version of pip causes installation issue with opencv
```
poetry run pip install --upgrade pip
```
Install dependencies
```
poetry install
```
Run the API
```
poetry run python app.py
```

Docker

MetaMorph API can be quickly deployed from docker without any installation or model download steps.

To use it, pull the image from dockerhub at vinothpandian/metamorph

docker pull vinothpandian/metamorph:latest

Run it with docker the following command

docker run -p 8000:8000 --name metamorph vinothpandian/metamorph:latest

Development

To retrain or further improve MetaMorph model

Installation

Install Tensorflow Object Detection API using this installation guide.
Copy the config file from configs folder or setup your own Tensorflow Object Detection API Model with config
Download the UISketch dataset from Kaggle.
Generate synthetic data
```
poetry run python syn_datagen.py -d /path/to/uisketch -o /path/to/tfod/object_detection -l no.of.sketches
```
To check more options on data generation use poetry run python syn_datagen.py --help
Follow the guide from training and evaluation guide by TFOD team.

Citation

If you use MetaMorph, please use the following citation:

V. P. S. Pandian, S. Suleri, C. Beecks, M. Jarke. MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities. Proceedings of the 32st Australian Conference on Human-Computer-Interaction.

@inproceedings{Pandian_MetaMorph,
	title        = {MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities.},
	author       = {Pandian, Vinoth Pandian Sermuga and Suleri, Sarah and Beecks, Christian and Jarke, Matthias},
	year         = 2020,
	booktitle    = {Proceedings of the 32st Australian Conference on Human-Computer-Interaction},
	publisher    = {Association for Computing Machinery},
	address      = {New York, NY, USA},
	series       = {OZCHI'20},
	doi          = {10.1145/3441000.3441030},
	isbn         = {978-1-4503-8975-4/20/12},
	url          = {https://doi.org/10.1145/3441000.3441030},
	numpages     = 10,
	keywords     = {UI Element Dataset, Neural Networks, Deep Learning, Sketch Detection, Sketch Recognition, Artificial Intelligence, Prototyping}
}

If you use Syn or UISketch, please use the following citation:

@inproceedings{Pandin_Syn,
	title        = {Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches},
	author       = {Pandian, Vinoth Pandian Sermuga and Suleri, Sarah and Jarke, Matthias},
	year         = 2020,
	booktitle    = {Proceedings of the 25th International Conference on Intelligent User Interfaces Companion},
	location     = {Cagliari, Italy},
	publisher    = {Association for Computing Machinery},
	address      = {New York, NY, USA},
	series       = {IUI '20},
	pages        = {79–80},
	doi          = {10.1145/3379336.3381498},
	isbn         = 9781450375139,
	url          = {https://doi.org/10.1145/3379336.3381498},
	numpages     = 2,
	keywords     = {Neural Networks, Sketch Detection, Prototyping, Sketch Recognition, UI Element Dataset, Deep Learning}
}

Authors

👤 Vinoth Pandian

Website: vinoth.info
Github: @vinothpandian
LinkedIn: @vinothpandian

👤 Sarah Suleri

Website: sarahsuleri.info
LinkedIn: @sarahsuleri

vinothpandian/MetaMorph