Some tests failed on 0.4.0 due to low timeout
Closed this issue · 6 comments
xvitaly commented
Sometimes qcoroabstractsocket test fails (50/50) on 0.4.0:
+ /usr/bin/ctest --output-on-failure --force-new-ctest-process -j4 --exclude-regex test-qcoroprocess
Test project /builddir/build/BUILD/qcoro-0.4.0/release-qt5/redhat-linux-build
Start 1: test-qtimer
Start 2: test-qcorosignal
Start 3: test-task
Start 4: test-testconstraints
1/12 Test #4: test-testconstraints ............. Passed 0.01 sec
Start 5: test-qfuture
2/12 Test #2: test-qcorosignal ................. Passed 0.31 sec
Start 6: test-qdbuspendingcall
3/12 Test #1: test-qtimer ...................... Passed 0.72 sec
Start 7: test-qdbuspendingreply
4/12 Test #5: test-qfuture ..................... Passed 0.91 sec
Start 8: test-qcorolocalsocket
5/12 Test #6: test-qdbuspendingcall ............ Passed 1.12 sec
Start 9: test-qcoroabstractsocket
6/12 Test #3: test-task ........................ Passed 1.88 sec
Start 10: test-qcoronetworkreply
7/12 Test #7: test-qdbuspendingreply ........... Passed 3.12 sec
Start 11: test-qcorotcpserver
8/12 Test #11: test-qcorotcpserver .............. Passed 0.51 sec
Start 12: test-testhttpserver
9/12 Test #12: test-testhttpserver .............. Passed 1.52 sec
10/12 Test #8: test-qcorolocalsocket ............ Passed 5.05 sec
11/12 Test #10: test-qcoronetworkreply ........... Passed 4.52 sec
12/12 Test #9: test-qcoroabstractsocket .........***Failed 5.05 sec
********* Start testing of QCoroAbstractSocketTest *********
Config: Using QtTest library 5.15.2, Qt 5.15.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.2.1 20210728 (Red Hat 11.2.1-1)), fedora 35
PASS : QCoroAbstractSocketTest::initTestCase()
PASS : QCoroAbstractSocketTest::testWaitForConnectedTriggers()
PASS : QCoroAbstractSocketTest::testWaitForConnectedTimeout()
PASS : QCoroAbstractSocketTest::testWaitForDisconnectedTriggers()
PASS : QCoroAbstractSocketTest::testWaitForDisconnectedTimeout()
FAIL! : QCoroAbstractSocketTest::testDoesntCoAwaitConnectedSocket() No incoming connection within timeout!
Loc: [/builddir/build/BUILD/qcoro-0.4.0/tests/testhttpserver.h(99)]
PASS : QCoroAbstractSocketTest::testDoesntCoAwaitDisconnectedSocket()
PASS : QCoroAbstractSocketTest::testConnectToServerWithArgs()
PASS : QCoroAbstractSocketTest::testReadAllTriggers()
PASS : QCoroAbstractSocketTest::testReadTriggers()
PASS : QCoroAbstractSocketTest::testReadLineTriggers()
PASS : QCoroAbstractSocketTest::cleanupTestCase()
Totals: 11 passed, 1 failed, 0 skipped, 0 blacklisted, 5041ms
********* Finished testing of QCoroAbstractSocketTest *********
92% tests passed, 1 tests failed out of 12
Total Test time (real) = 6.48 sec
The following tests FAILED:
9 - test-qcoroabstractsocket (Failed)
Errors while running CTest
xvitaly commented
The same for qcorolocalsocket:
+ /usr/bin/ctest --output-on-failure --force-new-ctest-process -j4 --exclude-regex 'test-(qcoroprocess|qcoroabstractsocket)'
Test project /builddir/build/BUILD/qcoro-0.4.0/release-qt5/redhat-linux-build
Start 1: test-qtimer
Start 2: test-qcorosignal
Start 3: test-task
Start 4: test-testconstraints
1/11 Test #4: test-testconstraints ............. Passed 0.00 sec
Start 5: test-qfuture
2/11 Test #2: test-qcorosignal ................. Passed 0.31 sec
Start 6: test-qdbuspendingcall
3/11 Test #1: test-qtimer ...................... Passed 0.74 sec
Start 7: test-qdbuspendingreply
4/11 Test #5: test-qfuture ..................... Passed 0.91 sec
Start 8: test-qcorolocalsocket
5/11 Test #6: test-qdbuspendingcall ............ Passed 1.12 sec
Start 9: test-qcoronetworkreply
6/11 Test #3: test-task ........................ Passed 1.91 sec
Start 10: test-qcorotcpserver
7/11 Test #10: test-qcorotcpserver .............. Passed 0.51 sec
Start 11: test-testhttpserver
8/11 Test #7: test-qdbuspendingreply ........... Passed 3.12 sec
9/11 Test #11: test-testhttpserver .............. Passed 1.52 sec
10/11 Test #9: test-qcoronetworkreply ........... Passed 4.53 sec
11/11 Test #8: test-qcorolocalsocket ............***Failed 5.05 sec
********* Start testing of QCoroLocalSocketTest *********
Config: Using QtTest library 5.15.2, Qt 5.15.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.2.1 20210728 (Red Hat 11.2.1-1)), fedora 35
PASS : QCoroLocalSocketTest::initTestCase()
PASS : QCoroLocalSocketTest::testWaitForConnectedTriggers()
PASS : QCoroLocalSocketTest::testWaitForConnectedTimeout()
PASS : QCoroLocalSocketTest::testWaitForDisconnectedTriggers()
PASS : QCoroLocalSocketTest::testWaitForDisconnectedTimeout()
FAIL! : QCoroLocalSocketTest::testDoesntCoAwaitConnectedSocket() No incoming connection within timeout!
Loc: [/builddir/build/BUILD/qcoro-0.4.0/tests/testhttpserver.h(99)]
PASS : QCoroLocalSocketTest::testDoesntCoAwaitDisconnectedSocket()
PASS : QCoroLocalSocketTest::testConnectToServerWithArgs()
PASS : QCoroLocalSocketTest::testConnectToServer()
PASS : QCoroLocalSocketTest::testReadAllTriggers()
PASS : QCoroLocalSocketTest::testReadTriggers()
PASS : QCoroLocalSocketTest::testReadLineTriggers()
PASS : QCoroLocalSocketTest::cleanupTestCase()
Totals: 12 passed, 1 failed, 0 skipped, 0 blacklisted, 5040ms
********* Finished testing of QCoroLocalSocketTest *********
91% tests passed, 1 tests failed out of 11
Total Test time (real) = 5.96 sec
The following tests FAILED:
8 - test-qcorolocalsocket (Failed)
Errors while running CTest
Timeout should be significantly increased.
krop commented
Don't you see that because network is disabled in packaging build env?
xvitaly commented
No. The problem is very low timeout. Sometimes these tests pass, sometimes they don't.
krop commented
ok, you're right. From what I see in the openSUSE package (I'm preparing the update), these tests always fail on i586/x86_64 and randomly fail on PowerPC and ARM.
danvratil commented
There was a hidden race in the test, causing the test to sometimes terminate before the test server was able to register that something has happened. This is now synchronized properly, so the test should be stable now. Please reopen if you see some instabilities in the test suite again. Thanks! afa8824
krop commented
qdbuspendingcall still randomly fails with qcoro 0.6.0. I've seen the failure on armv7l and aarch64
[ 119s] 9/18 Test #10: test-qdbuspendingcall ............***Failed 1.20 sec
[ 119s] ********* Start testing of QCoroDBusPendingCallTest *********
[ 119s] Config: Using QtTest library 6.3.1, Qt 6.3.1 (arm-little_endian-ilp32-eabi-hardfloat shared (dynamic) release build; by GCC 12.1.0), unknown unknown
[ 119s] PASS : QCoroDBusPendingCallTest::initTestCase()
[ 119s] FAIL! : QCoroDBusPendingCallTest::testTriggers() 'shouldNotSuspend' returned FALSE. ()
[ 119s] Loc: [/home/abuild/rpmbuild/BUILD/qcoro-0.6.0/tests/testlibs/testobject.h(75)]
[ 119s] PASS : QCoroDBusPendingCallTest::testReturnsResult()
[ 119s] PASS : QCoroDBusPendingCallTest::testThenReturnsResult()
[ 119s] PASS : QCoroDBusPendingCallTest::testDoesntBlockEventLoop()
[ 119s] PASS : QCoroDBusPendingCallTest::testDoesntCoAwaitFinishedCall()
[ 119s] PASS : QCoroDBusPendingCallTest::cleanupTestCase()
[ 119s] Totals: 6 passed, 1 failed, 0 skipped, 0 blacklisted, 1145ms
[ 119s] ********* Finished testing of QCoroDBusPendingCallTest *********