octoprint crash caused by octoslack?
Closed this issue · 4 comments
Yesterday one of our octoprint installations crashed. The log file indicates problems from octoslack, however, I didn't have any time to investigate further...
20181120_ocotcrash.txt
Hi @uwezi
Thank you for attaching the OctoPrint log.
Octoslack may or may not be the culprit here.
The short version: Try turning off "Include RasPi temp" in the Octoslack config (it's at the top). Give that a try and let's see if your memory/crash issue comes back.
The longer version: Each time Octoslack calls out to the OS to grab the Pi's temperature, it uses the 'subprocess' library which perform a fork under the covers. While this is typically harmless, a fork is actually a rather intense process in that the OS is cloning the OctoPrint process (including it's allocated memory) and while some additional commands are then executed that likely free up that memory, the fork itself can be rather memory intensive. So - if OctoPrint itself (regardless of what Octoslack was doing) was consuming a decent amount of memory (either in context of the entire device or the memory limits imposed on the account running OctoPrint), it's conceivable that this logic to call out to get the Pi's temp could ask for more memory than the current user account/OS has available or will allow. Disabling the Octoslack option to check for the temp will avoid this process cloning/forking and that may 'fix' the issue. Beyond that I suspect there's a deeper issue of why your OctoPrint instance was consuming as much memory as it was - obviously this fork logic has been working for some time but we finally hit the breaking point where we weren't able to fork the current process again. It's entirely possible Octoslack itself is to blame (although I know a lot of people run it and I've yet to hear issues) or perhaps another plugin is to blame as well. Unfortunately it's going to be difficult to narrow down the root cause without some trial and error - disabling certain plugins, running plugins in a limited way (e.g. disable Snapshots in Octoslack).
As for Octoslack itself, I'm looking into a method to call out to the Pi temp command without needing the fork/high memory requirement. This isn't necessarily a fix but I feel it's generally a good idea all around.
Let me know what you find.
Good luck!
I myself run Octoslack on 4 printers and it was only this one which crashed so far... We previously had issues with the limitations of both Slack and Imgur and already disabled the snapshots - it simply doesn't work since there is no way to bulk-delete conveniently on either Imgur or Slack. Now on the same pi that crashed a cron-job is running deleting all Octoslack messages on Slack which are older than 24 hours, but this is a hazzle.
If there was a way to push snapshots to an FTP-service I would have the ability to host plenty of snapshots on my own server, with easy bulk-delete access as well...
It sounded like a nice option to check out the printers remotely, but...
Hi @uwezi
Any chance you've seen the memory issue occur again?
If so, we can dig in more. If not, I'll likely close out this issue until it occurs again.
As for the upload to ftp (or perhaps sftp), feel free to open a feature request and I'll likely work on it as time permits.
Closing due to inactivity; feel free to provide more details if you see the issue again.