cc-tweaked/CC-Tweaked

The big ol' compedium of flaky tests

SquidDev opened this issue · 0 comments

SquidDev commented

CC:T's test suite is incredibly flaky when run via CI. I think this is just a limitation of how powerful the runners are (they work fine on my machine!) - but having to rerun every third build gets old quickly.

This issue is an attempt to track recent tests failures.

Gametests fail due to timeout

Builds

Notes

In 2243, the test batch was created at 09:21:22 and finally started running at 09:21:25 (this is 3s for a single tick, which is Bad(TM)). Computers continue to turn on until 09:21:28, and then tests time out at 09:21:29 (after 150 ticks). There's two odd things here:

  • Computers really shouldn't take 3 seconds to all turn on - we're using the fake Kotlin machine here, which has very little startup overhead.
  • Tests time out after 150 ticks, which should take 7.5s, but only appears to be 4s. Unless Minecraft is skipping ticks here or something - maybe we should add some logging here just for debugging?

modem_test.have_peripherals.modem_test.have_peripherals

Starts with peripherals ==> 
 expected: <[monitor_0, printer_0, right]> 
  but was: <[right]> at 4,-58,22 (relative: 4,2,2) (t=1)

Builds

inventory_test.fails_on_full.inventory_test.fails_on_full

No peripheral: minecraft:furnace_0 at 47,-58,11 (relative: 2,2,1) (t=1)

Builds

modem_test.full_modems_form_networks.modem_test.full_modems_form_networks

On the same network ==> expected: <dan200.computercraft.impl.network.wired.WiredNetworkImpl@554b3d66> but was: <dan200.computercraft.impl.network.wired.WiredNetworkImpl@c4809f9>

Builds

turtle_test.peripheral_change.turtle_test.peripheral_change failed

expected: <[(peripheral_detach, right), (peripheral, right)]> 
 but was: <[(peripheral_detach, right)]>

Builds