Peerdiscovery tests are failing.

Question

Peerdiscovery tests are failing.

Closed this issue 10 months ago · 17 comments

The Jenkins job for bootstrap server liveness is failing:

[Test_BootstrapServers] $ /bin/sh -xe /tmp/jenkins[...].sh
+ /usr/bin/python3 py-ipv8/stresstest/bootstrap_rtt.py
Traceback (most recent call last):
  File "py-ipv8/stresstest/bootstrap_rtt.py", line 23, in <module>
    from ipv8.util import create_event_with_signals
ImportError: cannot import name 'create_event_with_signals' from 'ipv8.util' (/home/jenkins/.local/lib/python3.8/site-packages/ipv8/util.py)

We are now flying blind. This should be fixed ASAP.

EDIT: The bootstrap job is now running again. The peerdiscovery jobs are still broken.

Answer 1 · 2023-09-06T08:04:08.000Z

The same error appears for job/performance_peer_discovery and performance_peer_discovery_defaults.

Answer 2 · 2023-09-06T08:17:47.000Z

I tried wiping out the workspace but the job is not even running anymore. We may be out of disk space again.

Answer 3 · 2023-09-06T08:19:31.000Z

Actually, the tests are not running on #1176. This may be an even bigger issue.

Answer 4 · 2023-09-06T08:20:11.000Z

The badges have also disappeared from our readme.

Answer 5 · 2023-09-06T08:23:10.000Z

Maybe it's using the IPv8 from PyPI which doens't have this new function?

Answer 6 · 2023-09-06T08:24:39.000Z

I can confirm that the file exists with the appropriate function in the workspace.

At this point, I think its a Jenkins issue.

Answer 7 · 2023-09-06T08:31:36.000Z

Sure /home/jenkins/ci/workspace/ipv8/performance_peer_discovery/ipv8/util.py exists, but it seems to be looking for create_event_with_signals in /home/jenkins/.local/lib/python3.9/site-packages/ipv8/util.py.

Answer 8 · 2023-09-06T08:37:51.000Z

Ah, good catch. I guess someone installed IPv8 on bbq recently?

Regarding the badges disappearing: thanks @devos50 for cleaning up some experiment data. That part of the issue now seems solved.

Answer 9 · 2023-09-06T08:40:41.000Z

Judging by the logs, between 3 and 4 PM yesterday this failure started.

Answer 10 · 2023-09-06T08:48:19.000Z

Ah, it's not just bbq: himba-linux-builder has the same issue.

Answer 11 · 2023-09-06T08:51:59.000Z

If you set the Python path during the job, shouldn't everything be fine? Something like export PYTHONPATH=$(pwd):${PYTHONPATH}

Answer 12 · 2023-09-06T08:55:30.000Z

@egbertbouman that works, I'll do that for all failing jobs.

Answer 13 · 2023-09-06T08:59:48.000Z

Or try, I guess.. Jenkins is not having a good day today:

Answer 14 · 2023-09-06T09:09:08.000Z

Alright sitrep:

🆗 Badges
🆗 PR tests
🆗 Periodic Test_BootstrapServers
🚫 Periodic performance_peer_discovery: cannot apply configuration changes.
🚫 Periodic performance_peer_discovery_defaults: cannot apply configuration changes.

At least our bear minimum critical infrastructure is now operational. I'll scale this issue down to high.

Answer 15 · 2023-09-06T10:05:15.000Z

When it rains, it pours:

Answer 16 · 2023-09-06T14:40:01.000Z

According to StackOverflow the last two jobs being "unconfigurable" is because of outdated plugins. We would have to upgrade and restart Jenkins to fix this.

Answer 17 · 2023-09-07T13:05:44.000Z

Many thanks to @kozlovsky for figuring out which plugins needed an upgrade.

We are back up and operational! Thanks to everyone who helped fix this!