KTH/devops-course

Testing, verification and test automation

monperrus opened this issue · 51 comments

Wikipedia references:

"Automated testing is the act of conducting specific tests via automation (e.g. a set of regression tests) as opposed to conducting them manually, while test automation refers to automating the process of tracking and managing the different tests." source

Interesting tool.
Detailed failure reports and hassle free assertions
Scott test reporter
https://github.com/dodie/scott

From @AG00:

There is a relationship between desired properties (requirements), tests (verification) and test results. Those may be present in a hierarchy of components/subsystems/libraries. In addition, there is a relationship between planning activities on the system (product) being developed and its desired properties. Further, with time there will be more than one system/product configuration to consider and more than one version in use with different installations/deployments/customers. These relationships quickly becomes complex to understand and keep track of. As changes are being made (code or documents) and automatic tests are run, it would be good to have the information model changes generated rather than manually written. The information model can also be used to convince a customer (or the devops team itself) that a complex/high-level system property is achieved.

Testcontainers to test different environments on the developer's machine
https://www.testcontainers.org/

A post about the challenges to get global metrics for code coverage
https://massol.myxwiki.org/xwiki/bin/view/Blog/GlobalLocalCoverage

Security-based testing
https://www.inspec.io/

DeFlaker: Automatically Detecting Flaky Tests
https://www.jonbell.net/icse18-deflaker.pdf

Cross Browser Testing, Selenium Testing, and Mobile Testing
https://saucelabs.com/

Perfecto - Cloud-based Devops Testing
https://www.perfecto.io/

Automated Tool for Prioritizing JUnit Test Suites
https://github.com/kanonizo/kanonizo

mutation testing in JS
https://stryker-mutator.io/

Property-based testing in OCaml
https://github.com/c-cube/qcheck

@robertfeldt you may want to subscribe to this issue.

Amazon codeguru
It’s like having a distinguished engineer on call, 24x7
https://aws.amazon.com/codeguru/

New testing course from TU Delft
https://sttp.site/

Rultor is a DevOps team assistant. It helps your programmers and release managers automate routine operations (merge, deploy and release), with an easy-to-use intuitive chatbot interface.
https://github.com/yegor256/rultor

Jasmine is a behavior-driven development framework for testing JavaScript code.
https://jasmine.github.io/

Catch2: modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++
https://github.com/catchorg/Catch2

Commercial product to accelerate tests with an automated dependency graph
https://yourbase.io/

Automated Cucumber-JVM test pack for DECS built on the Serenity BDD framework
https://github.com/UKHomeOffice/hocs-serenity-automation

Testing at Electronic Arts
SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning
https://arxiv.org/pdf/2203.05566.pdf

Property-based testing in language X
For example, in javascript
https://marmelab.com/blog/2019/04/18/property-based-testing-js.html

An interesting essay / demo / tutorial topic: Automated migration of Junit4 to Junit5

Suggestion: Testing for GraphQL APIs and Engines.

An interesting and very original topic for a presentation / demo in software testing:
code coverage in production

Topic suggestion: in depth analysis of a mocking library in language X (eg Mockito)

fuzz testing is an essential topic for testing and security

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA). https://robotframework.org/

Predictive Test Selection cuts testing time up to 70% by using machine learning to select and run only tests that are likely to provide useful feedback during test runs
https://gradle.com/gradle-enterprise-solutions/predictive-test-selection/

Load-Test-As-Code. https://gatling.io/

Instancio is a Java library that automatically creates and populates objects for your unit tests.
https://github.com/instancio/instancio

Mutagen and cargo-mutants are two relevant mutation testing tools for Rust.

Credits @VIGGEEN
cc/ @ErikNatanael

A command-line benchmarking tool for rust
https://github.com/sharkdp/hyperfine

A library for generating fake data in Rust.
https://github.com/cksac/fake-rs

Integrity Test Framework - a sophisticated integration test automation framework for Java
https://github.com/integrity-tf/integrity

Newman is a command-line collection runner for Postman
https://github.com/postmanlabs/newman

Framework for web testing, and capture&replay
https://github.com/microsoft/playwright

Kurtosis is a composable build system for multi-container test environments.
https://docs.kurtosis.com/