buildtesters/buildtest

add code coverage for retrying tests (buildtest build --retry)

Closed this issue · 2 comments

shahzebsiddiqui commented

@Mendi03 i will be assigning this task to you

shahzebsiddiqui commented

@Mendi03 here are few hints for you to get started

The test method run will return if exit code is a 0 or when its run on a local executor because non-zero exit code tests should stop executing as pose to keep running again.

buildtest/buildtest/builders/base.py

Lines 378 to 379 in 6b6277c

if ret == 0 or self.is_local_executor():
return command

For instance if you look at this example test https://github.com/buildtesters/buildtest/blob/devel/tutorials/test_status/pass_returncode.yml simply running a exit 1 will cause test to fail. That being said we do need the or self.is_local_executor() in the if condition otherwise it will keep retrying.

I propose we change this if condition such that its something like so that we return immediately if buildtest build --retry is not specified and the section below would be logic for retry.

if not self._retry or ret == 0:
    return command

We should also change this line from 1 to None as the default.

buildtest/buildtest/builders/base.py

Line 111 in 6b6277c

self._retry = 1
.

There is a retry method that is used to get the value from the command line in this method

buildtest/buildtest/builders/base.py

Lines 326 to 327 in 6b6277c

def retry(self, retry):
self._retry = retry

In the main build.py for every builder we pass the value of the command line buildtest build --retry <NUM> to each builder if that makes sense.

buildtest/buildtest/cli/build.py

Line 1123 in 6b6277c

builder.retry(self.retry)

What this means we should expect the following

  1. If we invoke buildtest build without --retry the test should run once regardless if test pass or fails
  2. If --retry is specified we should only retry test that fails (i.e non-zero returncode) X number of times.

To add the regression test i would suggest you do the following