If you're tasked with the assessment of an ongoing test automation project, or if you're looking to start a new project and are looking for pointers on what to take into consideration, I've compiled a list of questions that can help you doing just that.
This is by no means an official list, nor do I claim that it's complete, but I do think that's the list is pretty extensive and covers a wide range of aspects involved with making a test automation project successful.
Here goes!
- Do you have an answer to the question of 'Why do we automate?' What are the goals you hope to achieve by implementing test automation?
- Is everybody on the same page with regards to the answer to the previous question?
- Do you think your current test automation efforts contribute to the goals you have set? If not, what do you think is the root cause of that?
- Do your test automation activities support your testing and development efforts in the way and to the extent that you expect or hoped for?
- Do you know where you want to be with your test automation six months to a year from now?
- Is there a clear roadmap, action plan or backlog that contains actionable tasks that contribute towards this desired result?
- Does your team or organization allot enough time and manpower for the implementation of these test automation goals (in sprint, as part of a project or otherwise)?
- Do the people responsible for creating and maintaining the automation have sufficient knowledge to perform these tasks?
- What does your approach to obtaining and sharing knowledge with regards to automation look like?
- Is the creation and maintenance of your automated tests an integral part of your development tasks / user stories / projects?
- Are automated tests run on a regular schedule, e.g. nightly or automatically triggered after every commit?
- Is your test code / are your automated tests subject to peer and/or code reviews? This applies to low-code automation as well as code-based automation
- Is your current test automation tool setup fit for purpose?
- How long does it take new team members to become comfortable in using and contributing to your current test automation setup?
- How easy is it to hire for new people that possess the skills necessary to work with your current test automation setup?
- Are your tests written at the lowest possible level (e.g., unit tests, integration tests)?
- Is the creation of the initial state for the tests (setup of environment and test data) and the cleaning up (teardown) handled efficiently (i.e., without relying on the user interface too much)?