sinnwerkstatt/runrestic

Recursion error

fdw opened this issue · 15 comments

fdw commented

I'm getting a lot of similar errors in different circumstances. They all look more or less like this:

Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/util.py", line 201, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 726, in _terminate_pool
    p.join()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 47, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 1 more time]
... snip ...
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 15 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Importantly, I have set

[execution]
parallel = false

I'm running 0.5.2 from the Arch AUR. What else do you need?

Do you have any idea what the problem could be?

fdw commented

With some more trial and error, I learned that runrestic backup works perfectly, but check and especially prune run into these errors, but also not always.

andreasnuesslein commented

Ahoy @fdw ,

sorry I completely forgot about this - stressful days as of late. Ergo I also didn't have any time to investigate this much :(

I'm assuming parallel = false is important for you?

Does the above error happen at the beginning of the run, or rather towards the end? I'm confused about the fact that it's erroring on os.killpg(os.getpgrp(), signal_number) which is only there to handle the case where the processes are killed/terminated basically.

Could you provide a full trace (with -l debug)?

fdw commented

sorry I completely forgot about this - stressful days as of late. Ergo I also didn't have any time to investigate this much :(

No worries :)

I'm assuming parallel = false is important for you?

No, not at all - I had just hoped that it would fix the problem, as it seems to be about multithreading.

Does the above error happen at the beginning of the run, or rather towards the end?

It varies a bit, but I would say rather soon-ish.

Could you provide a full trace (with -l debug)?

Spawning "['restic', '-r', '<snip>', 'check']"
Spawning "['restic', '-r', '<snip>', 'forget', '--keep-last', '1', '--keep-daily', '7', '--keep-weekly', '13', '--keep-monthly', '15', '--keep-yearly', '5', '--group-by', 'host,paths']"
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/util.py", line 416, in _flush_std_streams
Exception ignored in sys.unraisablehook: <built-in function unraisablehook>
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

In this case, the error happend almost instantly after the prune command was started.

andreasnuesslein commented

@fdw very odd. Could you attach the full config file? (strip the actual repo names of course)
Is it one repo or multiple?

fdw commented

Just one repo, locally.

Here's a full log:

Parsing configuration file: .config/restic/home.toml
[Environment] RESTIC_PASSWORD=**********
[Environment] B2_ACCOUNT_ID=<snip>
[Environment] B2_ACCOUNT_KEY=<snip>
Spawning "['restic', '-r', '<snip>', 'check']"
Spawning "['restic', '-r', '<snip>', 'forget', '--keep-last', '1', '--keep-daily', '7', '--keep-weekly', '13', '--keep-monthly', '15', '--keep-yearly', '5', '--group-by', 'host,paths']"
Exception ignored in sys.unraisablehook: <built-in function unraisablehook>
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 992 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
Spawning "['restic', '-r', '<snip>', 'prune']"
Exception ignored in: <Finalize object, dead>

During handling of the above exception, another exception occurred:

Exception ignored in sys.unraisablehook: <built-in function unraisablehook>
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 4 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 95 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 32 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 14 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 23 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 14 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 14 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 4 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 23 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 2 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 44 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 4 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 23 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 32 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 14 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 14 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 41 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 14 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 50 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 23 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 22 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 14 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 5 more times]
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 276 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
Exception ignored in: <function WeakSet.__init__.<locals>._remove at 0x7fb355415040>
Traceback (most recent call last):
  File "/usr/lib/python3.8/_weakrefset.py", line 38, in _remove
    def _remove(item, selfref=ref(self)):
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 992 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
{
  "check": {
    "<snip>": {
      "errors": 0,
      "errors_data": 0,
      "errors_snapshots": 0,
      "read_data": 0,
      "check_unused": 0,
      "duration_seconds": 3.3062503337860107,
      "rc": 0
    }
  },
  "forget": {
    "<snip>": {
      "removed_snapshots": 0,
      "duration_seconds": 0.7478501796722412,
      "rc": 0
    }
  },
  "prune": {
    "<snip>": {
      "containing_packs_before": "769",
      "containing_blobs": "87467",
      "containing_size_bytes": 3349000749.056,
      "duplicate_blobs": "0",
      "duplicate_size_bytes": 0.0,
      "in_use_blobs": "87467",
      "removed_blobs": "0",
      "invalid_files": "0",
      "deleted_packs": "0",
      "rewritten_packs": "0",
      "size_freed_bytes": 0.0,
      "removed_index_files": "1",
      "duration_seconds": 4.296233177185059,
      "rc": 0
    }
  },
  "last_run": 1584090712.671599,
  "total_duration_seconds": 8.497836351394653
}

Two notes:
First, the run directly before this one had no problems at all.
Secondly, the errors pop up asynchronously, even after runrestic was killed.

andreasnuesslein commented

Thanks for the full log - but i meant the config file :D

However: you use a B2 repo - could you check if the problem persists when you simply use a local directory instead (repositories = ["/tmp/restic-repo"]) ?

fdw commented

Oh, sorry. Here's the config:

repositories = [
  "/run/media/<snip>"
]

[environment]
RESTIC_PASSWORD = "<snip>"

[execution]
parallel = false
retry_count = 1

[backup]
sources = [
  "<snip>"
]

exclude_patterns = [
  <snip>
]

post_hooks = [
  'curl <snip>'
]

[prune]
keep-last = 1
keep-daily = 7
keep-weekly = 13
keep-monthly = 15
keep-yearly = 5
group-by = "host,paths"

[checks]
checks = ["check-unused", "read-data"]

I do not actually use a b2 repo but a local one, but I forgot the config parameters after some copy-paste action 😉 I removed them from the config just now, and the problem stays the same.

However, I also ran into this (similar, but not identical) output:

Spawning "['restic', '-r', '<snip>', 'check']"
Spawning "['restic', '-r', '<snip>', 'forget', '--keep-last', '1', '--keep-daily', '7', '--keep-weekly', '13', '--keep-monthly', '15', '--keep-yearly', '5', '--group-by', 'host,paths']"
Traceback (most recent call last):
Error in atexit._run_exitfuncs:

Spawning "['restic', '-r', '<snip>', 'prune']"
Error in atexit._run_exitfuncs:
Error in atexit._run_exitfuncs:
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fce04650040>
Traceback (most recent call last):
  File "/usr/lib/python3.8/weakref.py", line 103, in remove
    def remove(wr, selfref=ref(self), _atomic_removal=_remove_dead_weakref):
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 992 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
andreasnuesslein commented

Hmmmm... still very odd. Is your computer maybe maxing out CPU/RAM wise around that time? Does dmesg report an OOM-kill (on restic) after runrestic failed?

fdw commented

Nope, I can see nothing unusual :(

Other data points:

  • I can reproduce it on another, similarly configured machine.
  • It doesn't happen all the time, but most of the time (> 80%, I'd guess)
  • It mostly happens in the restic forget call, but I also managed to get it after check and probably after prune, too.
andreasnuesslein commented

I saw some python3.8 bug concerning weakref - although it was relevant to a beta of python3.8 I guess. Would it be possible for you to try pyenv with python 3.7 and see if the problem remains then?

I will try and pyenv to python3.8 and maybe I can reproduce the bug there then

fdw commented

Ha! With Python 3.7.6, it works beautifully (both version 0.5.2 and 0.5.3). So I guess that it's a Python 3.8 problem ;)

fdw commented

I played around a bit and found something that works for me. Hope you find this useful 🙂

andreasnuesslein commented

! Perfect 🙌 and thanks a bunch for the PR

fdw commented

Sure thing. Now I'm eagerly waiting for the next release 😉

andreasnuesslein commented

I bumped the version right after ;) - 0.5.4 is your version