This is the online appendix for the following paper submission:
Bakaev, M., Speicher, M., Jagow, J., Heil, S., Gaedke, M.: We Don’t Need No Real Users?! Surveying the Adoption of User-less Automation Tools by UI Design Practitioners. To appear in Proceedings of the 22nd International Conference on Web Engineering, ICWE 2022.
For details on the survey conducted, please refer to the Survey Questions and Survey Results.
The following table presents the 61 tools used as data source in our userless design survey. The descriptions provided are quotations or summaries taken from their corresponding references.
| Category | Tool | Description | Commercial/Academic | Reference |
|---|---|---|---|---|
| UI Prototyping Automation | Adobe XD | A powerful and easy-to-use vector-based experience design platform that gives teams the tools they need to craft the world’s best experiences collaboratively | commercial | https://www.adobe.com/products/xd.html |
| UI Prototyping Automation | Axure | The only UX tool that gives UX professionals the power to build realistic, functional prototypes. | commercial | https://www.axure.com/ |
| UI Prototyping Automation | Balsamiq | The industry standard low-fidelity wireframing software. | commercial | https://balsamiq.com/ |
| UI Prototyping Automation | Figma | Figma connects everyone in the design process so teams can deliver better products, faster. | commercial | https://www.figma.com/ |
| UI Prototyping Automation | InVision | The digital product design platform powering the world’s best user experiences | commercial | https://www.invisionapp.com/ |
| UI Prototyping Automation | JustInMind | All-in-one prototyping tool for web and mobile apps | commercial | https://www.justinmind.com/ |
| UI Prototyping Automation | Mockplus | The simplest yet powerful way to design, prototype and collaborate | commercial | https://www.mockplus.com/download/mockplus-classic |
| UI Prototyping Automation | Origami Studio | Design, animate, and prototype. All-in-one. | commercial | https://origami.design/ |
| UI Prototyping Automation | Sketch | The home for your entire collaborative design process. | commercial | https://www.sketch.com/ |
| UI Prototyping Automation | uizard.io | The design tool for non-designers. | commercial | https://uizard.io/ |
| Generative Tools based on AI/ML | GRIDS | A wireframing tool that provides designers with real-time layout suggestions. | academic | Niraj Ramesh Dayama, Simo Santala, Lukas Brückner, Kashyap Todi, Jingzhou Du, and Antti Oulasvirta. 2021. Interactive Layout Transfer. In 26th International Conference on Intelligent User Interfaces. ACM, New York, NY, USA, 70–80. https://doi.org/10.1145/3397481.3450652 https://userinterfaces.aalto.fi/grids/ |
| Generative Tools based on AI/ML | MenuOptimizer | MenuOptimizer assists in the design of menus. | academic | Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. MenuOptimizer: Interactive optimization of menu systems. In UIST 2013 - Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 331–341. https://doi.org/10.1145/2501988.2502024 https://www.gillesbailly.fr/menuoptimizer.html |
| Generative Tools based on AI/ML | UIZard Design Assistant | Turn sketches into editable designs. | commercial | https://uizard.io/design-assistant/ |
| Generative Tools based on AI/ML | Paper2Wire | Uses ML to match graphical user interface elements in sketches on paper to their digital counterparts to create consistent wireframes. | academic | Daniel Buschek, Charlotte Anlauff, and Florian Lachner. 2020. Paper2Wire: A Case Study of User-Centred Development of Machine Learning Tools for UX Designers. In Proceedings of the Conference on Mensch und Computer (Magdeburg, Germany) (MuC ’20). ACM, New York, NY, USA, 33–41. https://doi.org/10.1145/3404983.3405506 |
| Generative Tools based on AI/ML | Sketch2Code | Transform any hand-drawn design into a HTML code with AI. | other | https://sketch2code.azurewebsites.net/ |
| Generative Tools based on AI/ML | UI-image-to-GUI-skeleton | A neural machine translator that combines recent advances in computer vision and machine translation for translating a UI design image into a GUI skeleton. | academic | Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE’18). ACM, New York, NY, USA, 665–676. https://doi.org/10.1145/3180155.3180240 |
| Automated (no users) usability validation/evaluation | Qualidator | Test the quality of your website! | academic | https://www.qualidator.com/ |
| Automated (no users) usability validation/evaluation | Usability Smells Finder | Analyses user interaction events on-the-fly, discovers usability smells and reports them together with a concrete solution in terms of a usability refactoring, providing usability advice for deployed web applications. | academic | Julián Grigera, Alejandra Garrido, José Matías Rivero, and Gustavo Rossi. 2017. Automatic detection of usability smells in web applications. International Journal of Human-Computer Studies 97 (jan 2017), 129–148. https://doi.org/10.1016/j.ijhcs.2016.09.009 http://autorefactoring.lifia.info.unlp.edu.ar/ |
| Automated (no users) usability validation/evaluation | USEFul | A Framework to Mainstream Web Site Usability Through Automated Evaluation. | academic | Alexiei Dingli and Justin Mifsud. 2011. USEFul: A Framework to Mainstream Web Site Usability Through Automated Evaluation. International Journal of Human Computer Interaction (IJHCI) 2, 1 (2011), 10. |
| Automated (no users) usability validation/evaluation | WaPPU | A tool for realizing the novel concept of Usability-based Split Testing. | academic | Maximilian Speicher, Andreas Both, and Martin Gaedke. 2014. Ensuring Web Interface Quality through Usability-Based Split Testing. In Web Engineering, 14th International Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings (Lecture Notes in Computer Science, Vol. 8541), Sven Casteleyn, Gustavo Rossi, and Marco Winckler (Eds.). Springer, 93–110. https://doi.org/10.1007/978-3-319-08245-5_6 Maximilian Speicher, Andreas Both, and Martin Gaedke. 2015. S.O.S.: Does Your Search Engine Results Page (SERP) Need Help?. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1005–1014. https://doi.org/10.1145/2702123.2702568 |
| UI/UX-related metrics/KPIs | AIM | Compute how good your design is! | academic | Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, Gregorio Palmas, and Tino Weinkauf. 2018. Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings (Berlin, Germany) (UIST ’18 Adjunct). ACM, New York, NY, USA, 16–19. https://doi.org/10.1145/3266037.3266087 https://interfacemetrics.aalto.fi/ |
| UI/UX-related metrics/KPIs | Clicks/CTR prediction (e.g. UBR4CTR) | User Behavior Retrieval for CTR prediction | academic,commercial | Jiarui Qin, Weinan Zhang, Xin Wu, Jiarui Jin, Yuchen Fang, and Yong Yu. 2020. User Behavior Retrieval for Click-Through Rate Prediction. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, USA, 2347–2356. https://doi.org/10.1145/3397271 https://github.com/qinjr/UBR4CTR |
| UI/UX-related metrics/KPIs | Eye-gaze/RoI prediction (e.g. Eyequant) | We fuse AI and neuroscience to help you design seamless, conversion-boosting UX | academic,commercial | Zoya Bylinskii, Nam Wook Kim, Peter O’donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. 2017. Learning Visual Importance for Graphic Designs and Data Visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 57–69. https://doi.org/10.1145/3126594 https://www.eyequant.com/ |
| UI/UX-related metrics/KPIs | Web Vitals | Performance Metrics | other | An initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web. |
| UI/UX-related metrics/KPIs | Zyro Heatmaps | One of the AI-powered analytics tools that can help you understand how a web page looks like to your visitors. | commercial | https://zyro.com/ai/heatmap |
| Design guidelines organization/validation tools | Material Design guidelines | An adaptable system of guidelines, components, and tools that support the best practices of user interface design. | other | https://material.io/ |
| Design guidelines organization/validation tools | UIS Hunter | An automated UI design smell detector. | academic | https://uishunter.net.cn/ |
| Design guidelines organization/validation tools | Test.ai | Go Beyond Legacy Software Test Automation Tools with test.ai. | commercial | https://www.test.ai/ |
| Design guidelines organization/validation tools | XCUITest | Make sure that your app’s user interface behaves correctly when expected actions are performed. | commercial | https://developer.apple.com/documentation/xctest/user_interface_tests |
| GOMS/KLM tools | CogTool | A general purpose UI prototyping tool with a difference – it automatically evaluates your design with a predictive human performance model (a “cognitive crash dummy”). | academic | Rachel Bellamy, Bonnie John, and Sandra Kogan. 2011. Deploying CogTool. In Proceedings of the 33rd International Conference on Software Engineering. ACM, New York, NY, USA, 691–700. https://doi.org/10.1145/1985793.1985890 https://www.cogtool.org/ |
| GOMS/KLM tools | Cogulator | A simple human performance calculator developed by MITRE for estimating task time and difficulty (i.e., workload). | academic | https://cogulator.io/ |
| GOMS/KLM tools | KLM calculator | Card, Moran and Newell proposed a keystroke-level model that predicted the efficiency of an interface for an expert user. | other | https://syntagm.co.uk/design/klmcalc.shtml |
| HTML/CSS and accessibility validators | AChecker | A Web accessibility evalution tool designed to help Web content developers and Web application developers ensure their Web content is accessible. | academic | https://achecker.achecks.ca/checker/ |
| HTML/CSS and accessibility validators | Browser web analytics plugins (e.g. Lighthouse) | An open-source, automated tool for improving the quality of web pages. | other | https://developers.google.com/web/tools/lighthouse |
| HTML/CSS and accessibility validators | Level Access Web Accessibility Tools | Is your website accessible to people with disabilities? | commercial | https://www.webaccessibility.com/ |
| HTML/CSS and accessibility validators | SortSite | A one-click web site testing tool used by federal agencies, Fortune 100 corporations and independent consultancies. | commercial | https://www.powermapper.com/products/sortsite/ |
| HTML/CSS and accessibility validators | WAVE | A suite of evaluation tools that helps authors make their web content more accessible to individuals with disabilities. | academic | https://wave.webaim.org/ |
| HTML/CSS and accessibility validators | W3C validator | Checks the markup validity of Web documents in HTML, XHTML, SMIL, MathML, etc. | other | https://validator.w3.org/ |
| Behavior simulation/GUI testing automation | Eggplant | The industry leading test automation and monitoring platform that features deep learning and artificial intelligence to help your retail software scale and meet your customer's needs. | commercial | https://www.eggplantsoftware.com/ |
| Behavior simulation/GUI testing automation | Tricentis Tosca | Optimizes and accelerates end-to-end testing of your entire digital landscape. | commercial | https://www.tricentis.com/products/automate-continuous-testing-tosca/ |
| Behavior simulation/GUI testing automation | Appium | An open source test automation framework for use with native, hybrid and mobile web apps. | commercial | https://appium.io/ |
| Behavior simulation/GUI testing automation | iMacros | The world's most popular web automation, data extraction, and web testing solution – now with Chromium browser technology for supporting all modern websites. | commercial | https://www.progress.com/imacros |
| Behavior simulation/GUI testing automation | Katalon | Katalon is the leading low-code test automation solution for Web, API, Mobile, and Desktop apps. | commercial | https://www.katalon.com/ |
| Behavior simulation/GUI testing automation | Linux Desktop Testing Project | Cross Platform GUI Test Automation tool | other | https://ldtp.freedesktop.org/wiki/ |
| Behavior simulation/GUI testing automation | Robot Framework | A generic open source automation framework. | other | https://robotframework.org/ |
| Behavior simulation/GUI testing automation | Selenium | Selenium automates browsers. | other | https://www.selenium.dev/ |
| Behavior-driven development | Capybara | A library written in the Ruby programming language which makes it easy to simulate how a user interacts with your application. | other | https://teamcapybara.github.io/capybara/ |
| Behavior-driven development | Cucumber | A tool that supports Behaviour-Driven Development (BDD). | commercial | https://cucumber.io/ |
| Behavior-driven development | Ranorex Studio | Functional UI Test Automation | commercial | https://www.ranorex.com/ |
| Behavior-driven development | TestComplete | Automated UI testing that covers you from device cloud to packaged apps. | commercial | https://smartbear.com/product/testcomplete/overview/ |
| Behavior-driven development | Twist | Twist can help keep this task achievable for consumers of acceptance tests by allowing the definition of confirmation language to be natural. | commercial | http://studios.thoughtworks.com/twist-agile-test-automation/ |
| Behavior-driven development | Behat | A php framework for autotesting your business expectations. | other | https://behat.org/ |
| Model-driven development | Appian | Low-Code Automation | commercial | https://appian.com/ |
| Model-driven development | AXIOM | A model-driven approach for developing high quality, cross-platform appli- cations | academic | Xiaoping Jia and Chris Jones. 2013. Cross-Platform Application Development Using AXIOM as an Agile Model-Driven Approach. In ICSOFT 2012. Communications in Computer and Information Science, Vol. 411. Springer Berlin Heidelberg, 36–51. https://doi.org/10.1007/978-3-642-45404-2_3 |
| Model-driven development | CaseComplete | The agile requirements management tool that Business Analysts love. | commercial | https://casecomplete.com/ |
| Model-driven development | IBM Rational Rhapsody | A proven solution for modeling and design activities that helps you deliver higher quality systems and software faster | commercial | https://www.ibm.com/products/systems-design-rhapsody |
| Model-driven development | Mendix Studio | Second-generation of Mendix Assist, industry-first AI engine for no-code/low-code, ratchets up productivity and quality | commercial | https://www.mendix.com/ |
| Model-driven development | WebRatio | The Low-Code Development Platform for Digital Transformation | commercial | https://www.webratio.com/ |
| Website visual appearance | ViCRAM tool | Predicts and visualises the complexity of web pages in the form of a pixelated heat map. | academic | Eleni Michailidou, Sukru Eraslan, Yeliz Yesilada, and Simon Harper. 2021. Automated prediction of visual complexity of web pages: Tools and evaluations. International Journal of Human-Computer Studies 145 (jan 2021), 102523. https://doi.org/10.1016/j.ijhcs.2020.102523 |
| Website visual appearance | VisualMind AI | Instantly analyse how many people will like your website and more... | commercial | https://myraah.io/visualmind |
| Website visual appearance | Color Contrast checkers (e.g. colourcontrast.cc) | Check the contrast between different colour combinations against WCAG standards | other | https://colourcontrast.cc/ |