Deepstack Windows 2020.12 Beta - 404 error

[VorlonCD]

"GPU" version 2020.12:

After a day or so of running fine with AITOOL, the Deepstack url (http://localhost:85/v1/vision/detection in my case) starts giving a 404 page not found error, even in a browser.

This file was created about when it happened:
C:\DeepStack\redis\dump.rdb

And this was recently modified:
C:\Users\username\AppData\Local\DeepStack\logs\stderr.txt

As far as I can tell, the EXE's are all still running normally:

Just using built in Defender antivirus. I am running the latest Dev channel release of Windows 10 if it could be a factor.

I had closed/reopened AITOOL a little while before I noticed the error, so its at least possible that it was in the middle of a request to deepstack and the connection was abruptly disconnected.

Logs attached.
Thanks!
logs.zip

[johnolafenwa]

Hello @VorlonCD , thank you for sharing this. Have your tried restarting DeepStack? It appears DeepStack is unable to write its data to the AppData directory. This might be due to something on the windows end, probably an update. Restarting should fix this issue

[VorlonCD]

Yes, a restart of deepstack always fixes. I've seen 404 and 500 errors after it is running well for some time. I wonder if there is some resiliency that could be built in so the service restarts what it needs to get back on track after the error? A few theories why it might happen - 1) an abrupt disconnect in the middle of processing an image, 2) Windows cleans the TEMP folder and maybe it kills a temp file you expect to see?

[Tinbum1]

I've also seen the 500 error after running for about 2 days.

[VorlonCD]

@johnolafenwa - The attached update to detection.py does 2 things:

• Fixes issue where python.exe would crash due to unhanded crash in a FINALLY block
• Outputs actual exception message when the server returns error 500 - error occurred on the server. That way we get to see all the CUDNN_STATUS_ALLOC_FAILED and CUDA out of memory messages back in any program that reads the json ERROR property such as AITOOL. (latest commit, not released yet).

Detail:

I would commonly notice that Deepstack.exe did not have any spawned python.exe's's because they crashed.

As the STDERR.TXT shows:

File "C:\DeepStack\intelligencelayer\shared\detection.py", line 138, in objectdetection
os.remove(img_path)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\Users\\Vorlon\\AppData\\Local\\Temp\\DeepStack\\83e9c5b0-d698-44f3-a8df-d19655d9f7da

So for some reason (perhaps because the temporary image file is still 'in use') the os.remove command here gives that error:

finally:
                    db.set(req_id, json.dumps(output))
                    if os.path.exists(img_path):
                        os.remove(img_path)

And the exception is not caught because it is in a FINALLY block so the py script crashes bubbye.

The solution in the attachment calls a SafeFileDelete function that ignores the exception if it happens.

To use, backup and replace this file with the one in the zip.
C:\DeepStack\intelligencelayer\shared\detection.py

detection.zip

`