Flank/flank

Smart flank uses time of skipped test when test are added back

kozaxinan opened this issue · 0 comments

Describe the bug

A clear and concise description of what the bug is.

When a previously ignored test included in the test suite by removing @ignore annotation (TestA), smart flank does not use default test time. We noticed that TestA is included in first shard. When we have multiple tests added to test suit, all of them end up in first shard.
We noticed this situation because when we work on skipped tests, our first shard gets timeout. We expect previously skipped test should use default test time and probably end up in different shard, maybe new shards for all of new tests.

To Reproduce

Steps to reproduce the behavior:

  1. Have multiple tests including ignored tests.
  2. Setup smart flank. Set default test time and shard time same. Any new test should create new shard.
  3. Run a test suit with smart flank
  4. Remove @ignore annotation
  5. Run same tests with smart flank
  6. Previously ignored test ends up in first shard even though default test time is same as shard time.

Expected behavior

A clear and concise description of what you expected to happen.

Expect previously ignored test uses default test time and run in a new shard.

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

Post the output of flank --version.

Flank v23.03.2
Fladle 0.17.4
Android project