Making calls to external API with only 1 gunicorn worker
Closed this issue · 12 comments
Left a comment on a separate issue but realized it was already closed. Just to be clear, when I run gunicorn --worker-class eventlet -w 1 app:app, there is only one worker so I can't make calls to external APIs? I am trying to save my chat to a Firebase database and whenever I try to do so my gunicorn has a WORKER TIMEOUT.
The eventlet worker can handle multiple requests at once.
Your problem could be that you haven't monkey patched the Python standard library so that it is compatible with eventlet. Gunicorn normally does that, but I believe that is currently broken, so you should monkey patch in your application.
Hi Miguel,
Thank you so much for the help and I am sorry for the multiple posts. Thank you also for being so helpful in this community. It is amazing.
I am still having trouble even when I monkey patch.
Below is the top of my app.py:
`# import eventlet
eventlet.monkey_patch()
import gevent.monkey
gevent.monkey.patch_all()
async_mode = 'gevent'
import os
import re
import csv
import requests
from flask import Flask, request, jsonify, json, current_app, render_template, send_file, make_response, redirect
import json
import math
import secrets
import datetime
import jwt
from threading import Thread
from firebase_admin import credentials, firestore, initialize_app
import time
import hashlib
from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import Mail
from flask_socketio import SocketIO, join_room, leave_room, send
import uuid
from gevent import monkey
Initialize Firestore DB
cred = credentials.Certificate('key.json')
default_app = initialize_app(cred)
db = firestore.client()
Users = db.collection('Users')
WebsiteAnalytics = db.collection('Website Analytics')
Chats = db.collection('Chats')
app = Flask(name)
app.config['SECRET_KEY'] = 'gjr39dkjn344_!67#'
socketio = SocketIO(app, engineio_logger=True, logger=True, message_queue='redis://', async_mode=async_mode)`
The logs on my server whenever I try an endpoint with a Firebase call is so:
(env) 253-3:crush jamesbaker$ gunicorn --error-logfile - --worker-class gevent -w 1 app:app [2020-03-29 12:04:28 -0400] [30435] [INFO] Starting gunicorn 20.0.4 [2020-03-29 12:04:28 -0400] [30435] [INFO] Listening at: http://127.0.0.1:8000 (30435) [2020-03-29 12:04:28 -0400] [30435] [INFO] Using worker: gevent [2020-03-29 12:04:28 -0400] [30438] [INFO] Booting worker with pid: 30438 Server initialized for gevent. [2020-03-29 12:05:14 -0400] [30435] [CRITICAL] WORKER TIMEOUT (pid:30438) [2020-03-29 12:05:14 -0400] [30438] [INFO] Worker exiting (pid: 30438) [2020-03-29 12:05:14 -0400] [30445] [INFO] Booting worker with pid: 30445 Server initialized for gevent.
Same goes when I use eventlet:
(env) 253-3:crush jamesbaker$ gunicorn --error-logfile - --worker-class eventlet -w 1 app:app [2020-03-29 12:01:09 -0400] [30421] [INFO] Starting gunicorn 20.0.4 [2020-03-29 12:01:09 -0400] [30421] [INFO] Listening at: http://127.0.0.1:8000 (30421) [2020-03-29 12:01:09 -0400] [30421] [INFO] Using worker: eventlet [2020-03-29 12:01:09 -0400] [30424] [INFO] Booting worker with pid: 30424 Server initialized for eventlet. [2020-03-29 12:01:53 -0400] [30421] [CRITICAL] WORKER TIMEOUT (pid:30424) [2020-03-29 12:01:53 -0400] [30424] [INFO] Worker exiting (pid: 30424) [2020-03-29 12:01:53 -0400] [30430] [INFO] Booting worker with pid: 30430 Server initialized for eventlet.
Any idea what it could be? I have been trying to debug for a while now.
So when I printed eventlet.patcher.is_monkey_patched(firestore) it returned false. So maybe eventlet isn't monkey_patching my firebase module?
I don't think there is a lot to say about it. The monkey patching replaces the functions that are incompatible with eventlet with versions that are compatible. It's better to use packages that support greenlets natively, but for the vast majority of applications monkey patching is the only way, since there aren't a lot of packages written specifically for greenlets.
Found your comment from a separate thread^
Is it fair to assume that Firebase is not compatible with gunicorn and eventlet/gevent?
The monkey patching is for the standard library, there is no specific monkey patching for third-party packages, other than a few exceptions. Looking at firebase, it is sadly depends on Google SDKs for Python, which sadly do not work well with eventlet/gevent. See grpc/grpc#15923.
Makes sense! Thank you for all the help.
Noting somewhere that eventlet/gevent only is compatible with a certain set of Python packages may be useful for other users.
This is not something specific to eventlet and gevent, it is a characteristic of all async frameworks. It is true also for asyncio.
If you have any suggestions about how to include this let me know. I don't see how to explain how to use an async framework in a short note.
Nowhere under running with gunicorn does it mention monkey patching or the fact that it is not a panacea for all async issues when running with gunicorn. It just would be worth mentioning that considering it has taken a non-zero amount of time from myself along with others who I saw run into this problem elsewhere - especially considering when the project works without gunicorn just fine. Thank you for all that you do for the flask and socketio communities.
https://github.com/miguelgrinberg/Flask-SocketIO/blob/master/docs/index.rst#gunicorn-web-server
Gunicorn is currently broken on this respect, it should monkey patch for you. Your issue is unfortunate, but under normal circumstances you would have not seen problems related to no monkey patching.
Also as I was saying, async frameworks are very complex, you can't just say "monkey patch" and the problems go away. For many situations monkey patching makes things worse. Sometimes you need to monkey patch some functions, but leave others alone. Sometimes monkey patching does not solve your problem. I could add a note to cover your own case, but there are about a million other cases that are completely different than yours for which the advice given to you would be wrong and misleading.
You don't think it would be useful to note that Gunicorn is not currently correctly monkey patching and that it doesn't work perfectly for some "million other cases"?
No. Monkey patching isn't specific to gunicorn, so relying on gunicorn's automatic patching isn't a good idea anyway. If your application needs monkey patching you will want to monkey patch on your own, at the very least so that you can also run your app in development, where more than likely you will not be using gunicorn. So I do not believe documenting bugs in gunicorn wrt monkey patching in my documentation is necessary.
There are already several references to monkey patching in the documentation. Have you seen them? If you have suggestions for more, feel free to suggest where in a PR.
Okay, you win Miguel. I just hope someone who has this same issue sees this post in a timely manner and doesn't waste their time as I did.