Internal exception while trying to build Jenkins pipeline

[Ferruck]

Error occurred while trying to build BobBuildTool/basement#134 and BobBuildTool/basement#132, see the console log

Could be a configuration issue, nevertheless it should probably be handled properly.

An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Bob version 0.20.0
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/bob/scripts.py", line 153, in catchErrors
    ret = fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/bob/scripts.py", line 254, in cmd
    ret = cmd(args.args, bobRoot)
  File "/usr/local/lib/python3.9/dist-packages/bob/scripts.py", line 29, in __develop
    doDevelop(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/bob/cmds/build/build.py", line 314, in doDevelop
    commonBuildDevelop(parser, argv, bobRoot, True)
  File "/usr/local/lib/python3.9/dist-packages/bob/cmds/build/build.py", line 142, in commonBuildDevelop
    with EventLoopWrapper() as (loop, executor):
  File "/usr/local/lib/python3.9/dist-packages/bob/utils.py", line 702, in __init__
    self.__executor = getProcessPoolExecutor()
  File "/usr/local/lib/python3.9/dist-packages/bob/utils.py", line 688, in getProcessPoolExecutor
    executor.submit(dummy).result()
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 697, in submit
    self._adjust_process_count()
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 675, in _adjust_process_count
    p.start()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.9/multiprocessing/context.py", line 291, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.9/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.9/multiprocessing/popen_forkserver.py", line 51, in _launch
    self.sentinel, w = forkserver.connect_to_new_process(self._fds)
  File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 84, in connect_to_new_process
    self.ensure_running()
  File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 138, in ensure_running
    listener.bind(address)
OSError: [Errno 98] Address already in use

[jkloetzke]

Damn, this is caused by the light weight sandbox that I added on ci.bobbuildtool.dev to build pull requests. It executes bob in a new mount- and pid-namespace. This should isolate the build sufficiently from Jenkins itself and the processes that are started by it.

It turns out the multiprocessing module creates unix domain sockets in the abstract namespace (a Linux extension) and uses the PID for the name. Because each pull request is built in a separate PID namespace the PIDs will be the same. But we do not use a network namespace and all the unix domain sockets will be created in the same namespace -> boom. So as soon as more than one pull request is built all but the first one fill fail.

Right now I don't see how to solve this except using more heavy weight container solution...

[Ferruck]

Could you give some insights how you create that light-weight sandbox?

[jkloetzke]

I use unshare to create a pid namespace and then bob-namespace-sandbox to create a mount namespace that has just the workspace mounted writeable...