Slow execution of `DiscoveryBooster.take_step` due to frequent calls to `netifaces.ifaddresses`

[kozlovsky]

The DiscoveryBooster.take_step() method intermittently exhibits slow performance, sometimes taking several seconds to execute a single call, blocking the asyncio loop. This method internally invokes EdgeWalk.take_step(), where EdgeWalk is instantiated with the parameters neighborhood_size=25, edge_length=25.

I profiled the slow EdgeWalk.take_step() execution and discovered that the bulk of the execution time was consumed by calls to netifaces.ifaddresses and netifaces.interfaces. Here are the statistics derived from profiling:

50332 function calls in 2.015 seconds
Ordered by: internal time

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   580    1.577    0.003    1.577    0.003 {built-in method netifaces.ifaddresses}
   145    0.380    0.003    0.380    0.003 {built-in method netifaces.interfaces}
   150    0.007    0.000    0.008    0.000 tribler\venv\lib\site-packages\libnacl\__init__.py:506(crypto_sign)
   189    0.006    0.000    0.009    0.000 py-ipv8\ipv8\peerdiscovery\network.py:219(get_verified_by_address)
   105    0.005    0.000    0.005    0.000 {method 'acquire' of '_thread.RLock' objects}
   150    0.004    0.000    0.004    0.000 {method 'sendto' of '_socket.socket' objects}
   725    0.003    0.000    1.965    0.003 py-ipv8\ipv8\messaging\interfaces\endpoint.py:171(_get_interface_addresses)
   145    0.002    0.000    0.002    0.000 {built-in method builtins.__build_class__}
  9765    0.002    0.000    0.002    0.000 py-ipv8\ipv8\peer.py:72(addresses)
   450    0.001    0.000    0.003    0.000 py-ipv8\ipv8\messaging\serialization.py:353(pack_serializable)
   580    0.001    0.000    0.003    0.000 py-ipv8\ipv8\messaging\interfaces\endpoint.py:185(__init__)
   145    0.001    0.000    0.001    0.000 {method 'getsockname' of '_socket.socket' objects}
   150    0.001    0.000    1.988    0.013 py-ipv8\ipv8\community.py:225(create_introduction_request)
   145    0.001    0.000    1.969    0.014 py-ipv8\ipv8\messaging\interfaces\endpoint.py:299(_get_lan_address)
     1    0.001    0.001    2.015    2.015 py-ipv8\ipv8\peerdiscovery\discovery.py:110(take_step)
   580    0.001    0.000    0.001    0.000 py-ipv8\ipv8\messaging\interfaces\endpoint.py:217(<listcomp>)
  9770    0.001    0.000    0.001    0.000 {method 'values' of 'dict' objects}
  1175    0.001    0.000    0.001    0.000 {built-in method _socket.inet_aton}
   145    0.001    0.000    1.971    0.014 py-ipv8\ipv8\community.py:204(my_preferred_address)
  3096    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
   150    0.001    0.000    0.001    0.000 Python39\lib\ctypes\__init__.py:48(create_string_buffer)
   150    0.001    0.000    0.014    0.000 py-ipv8\ipv8\lazy_community.py:233(_ez_pack)
  1160    0.001    0.000    0.001    0.000 {built-in method _struct.unpack_from}
   150    0.001    0.000    1.994    0.013 py-ipv8\ipv8\community.py:461(walk_to)
   435    0.000    0.000    0.001    0.000 py-ipv8\ipv8\messaging\serialization.py:195(pack)
  2212    0.000    0.000    0.001    0.000 {built-in method builtins.isinstance}

As seen, the total execution time for EdgeWalk.take_step() was 2.015 seconds. Of this, 1.577 seconds were spent in netifaces.ifaddresses and 0.380 seconds in netifaces.interfaces.

Even though each individual invocation of netifaces.ifaddresses or netifaces.interfaces was reasonably fast (0.003 seconds per call), the sheer quantity of these calls (580 calls to netifaces.ifaddresses and 145 to netifaces.interfaces) resulted in a significant accumulation of execution time.

Proposed Mitigation:

To enhance the performance of the take_step method, we can cache the results from netifaces.ifaddresses and netifaces.interfaces. The cache duration could be a few seconds, which should significantly boost the speed of the take_step calls while still maintaining network information accuracy.

[qstokkink]

This is fixed in IPv8 release 2.11. The workaround fro this issue can be removed when the IPv8 dependency version is next updated.