openwisp/django-netjsongraph

test_multiple_receive_removed fails once in a while

nemesifier opened this issue · 5 comments

The test test_multiple_receive_removed in django_netjsongraph.tests.test_topology.TestTopology fails but only once every n test runs, where n may be 20/50/100, I don't know.

Here's how to reproduce it, this shell script runs the tests until it breaks (yeah, WTF).

while true
do
    ./manage.py test --keepdb --failfast django_netjsongraph.tests.test_topology
    if [ "$?" != "0" ]; then
        break
    fi
done

Likely has to do with mutable data structures, but it's just speculation.

Could you provide more details? (like package versions) I had been running that test in a loop for an hour without a fail.

@pawelplsi yeah same here, I also had no fail

@pawelplsi @Weirdo914 I got it (after 2 hours of loop working)

Screenshot from 2019-12-22 21-20-16

I have made several measurements which showed that this issue can be sometimes caused by lag spikes, especially in database queries, these lag spikes can cause tested links to expire before they are supposed to (expire time is 0.1s, the latency can sometimes be as high).
I think, to avoid such issues, we should avoid using directly functions like sleep() in tests.
In this PR: #105, I have implemented mock-tests using the freezegun library to make them more reliable and independent of the machine load.