timeout does not seem to work

Question

timeout does not seem to work

Opened this issue 3 years ago · 10 comments

kjappelbaum commented 3 years ago

@lpatiny reported that there seem to be issues with the timeout

Answer 1 · 2021-11-09T08:39:11.000Z

hmm ... i can provoke timeouts

Answer 2 · 2021-11-11T06:07:15.000Z

This morning I have again a lot of processes running

I will send you the full log by PM.

Answer 3 · 2021-11-11T06:08:46.000Z

Maybe the could be a route that shows the currently running processes and when they started ? Also are we sure that when there is a time out the process that makes the calculations is really killed ?

Answer 4 · 2021-11-11T06:10:58.000Z

There are new variables for the max number of processes and OMP thread. But here the issue is the webserver itself. We can simply limit it to two workers, shorten the backlog and also constrain the resources of the container On 11 Nov 2021, at 07:08, Luc Patiny ***@***.***> wrote: Maybe the could be a route that shows the currently running processes and when they started ? Also are we sure that when there is a time out the process that makes the calculations is really killed ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#17 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH3I2QNL3LERVJZZR5ZLBWLULNMXRANCNFSM5HUUYZXQ>.

Answer 5 · 2021-11-11T06:12:22.000Z

For information here is the list of processes currently running

root     16673  0.0  0.0   2376   504 ?        Ss   Nov09   0:00 /bin/sh -c gunicorn -w 4 xtbservice.xtbservice:app -b 0.0.0.0:$PORT -k uvicorn.workers.UvicornWorker
root     16743  0.0  0.0  31448 21728 ?        S    Nov09   0:20 /opt/conda/bin/python /opt/conda/bin/gunicorn -w 4 xtbservice.xtbservice:app -b 0.0.0.0:8091 -k uvicorn.workers.UvicornWorker
root     16854  0.0  0.1 4832068 121324 ?      Sl   Nov09   1:41 /opt/conda/bin/python /opt/conda/bin/gunicorn -w 4 xtbservice.xtbservice:app -b 0.0.0.0:8091 -k uvicorn.workers.UvicornWorker
root     16855  0.0  0.1 5366344 127272 ?      Sl   Nov09   1:42 /opt/conda/bin/python /opt/conda/bin/gunicorn -w 4 xtbservice.xtbservice:app -b 0.0.0.0:8091 -k uvicorn.workers.UvicornWorker
root     16864  0.0  0.1 6498408 147200 ?      Sl   Nov09   1:46 /opt/conda/bin/python /opt/conda/bin/gunicorn -w 4 xtbservice.xtbservice:app -b 0.0.0.0:8091 -k uvicorn.workers.UvicornWorker
root     24088 2330  0.1 3936608 105468 ?      Rl   Nov10 27220:13 /opt/conda/bin/python /opt/conda/bin/gunicorn -w 4 xtbservice.xtbservice:app -b 0.0.0.0:8091 -k uvicorn.workers.UvicornWorker
root     25707  0.0  0.0 112684   996 pts/4    S+   06:11   0:00 grep --color=auto gunicorn
root     25824  0.0  0.1 6176272 137140 ?      Sl   Nov10   0:51 /opt/conda/bin/python /opt/conda/bin/gunicorn -w 4 xtbservice.xtbservice:app -b 0.0.0.0:8091 -k uvicorn.workers.UvicornWorker
root     27536  0.0  0.0   2376    32 ?        Ss   Sep17   0:00 /bin/sh -c gunicorn -w 2 --backlog 16 dimensionality_reduction.dimensionality_reduction:app -b 0.0.0.0:$PORT -k uvicorn.workers.UvicornWorker
root     28464  0.0  0.0  33088  2344 ?        S    Sep17  10:52 /usr/local/bin/python /usr/local/bin/gunicorn -w 2 --backlog 16 dimensionality_reduction.dimensionality_reduction:app -b 0.0.0.0:14101 -k uvicorn.workers.UvicornWorker
root     30893  0.1  0.0 2567200 50356 ?       Sl   Sep17  88:04 /usr/local/bin/python /usr/local/bin/gunicorn -w 2 --backlog 16 dimensionality_reduction.dimensionality_reduction:app -b 0.0.0.0:14101 -k uvicorn.workers.UvicornWorker
root     30895  0.1  0.0 2788388 54800 ?       Sl   Sep17  87:45 /usr/local/bin/python /usr/local/bin/gunicorn -w 2 --backlog 16 dimensionality_reduction.dimensionality_reduction:app -b 0.0.0.0:14101 -k uvicorn.workers.UvicornWorker

Answer 6 · 2021-11-11T06:44:35.000Z

Maybe the could be a route that shows the currently running processes and when they started ?

typically this is nothing that i'd put in the service code

Answer 7 · 2021-11-11T06:45:08.000Z

when there is a time out the process that makes the calculations is really killed ?

this is the only thing i can imagine now. That there is still some thread remaining

Answer 8 · 2021-11-11T06:49:15.000Z

and you can also add cpu limit in the docker-compose to avoid that you manually need to restart https://docs.docker.com/compose/compose-file/compose-file-v2/#cpu-and-other-resources. Maybe it is easier if you just use v2

Answer 9 · 2021-11-11T07:05:25.000Z

i'll push an example with those settings

for me, locally, i couldn't find a way to run out of resources

Answer 10 · 2021-11-11T07:08:07.000Z

so i really started a lot concurrent requests