Vague Error Message if Server Overburdened
Closed this issue · 5 comments
The server had %Cpu(s): 99.5 us, 0.5 sy and Shiny applications weren't loading but the HTML landing page was. I suspended some processes using kill -STOP 12345 and then applications loaded fine again. The error message shown in Shiny Server is:
[2022-06-20 11:15:47.236] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2022-06-20 11:15:47.240] [ERROR] shiny-server - { Error: ENOENT: no such file or directory, open '/users/ststaff/shiny/logs/scClassify-shiny-20220620-111455-38221.log'
Could the error message be more informative about why a worker wasn't able to be started?
Seems to be independent of server load. It seems to relate to a Shiny app that has lots of package dependencies to load. R process in top uses 100% CPU for 50 seconds and then error appears, even though app_init_timeout is 300 in server.conf.
I would guess that R process is also very memory intensive and the process is being killed by the Linux OOM killer. Do you see the memory climbing before it’s killed?
No, it's low memory usage and the server happens to have 512 GB physical RAM so it never reaches close to disk swapping. After all, it's just loading a list of packages and never gets to a useable interface for the user to do any processing of data with.
Perhaps try xrprof to see what that R process is doing. Any code outside the server function is going to execute at process startup, in some apps that’s minutes of work (not recommended of course).
Was this app ever working?
It was failing most of the time yesterday. Today, it works every time. I can't reproduce the error. I'll remember to try xrprof.