Running Pyramid on ASGI
Natim opened this issue · 27 comments
You may have noticed that the Django community started a project called channels which aims to provide a way to handle WebSocket and the HTTP/2 protocol with Python workers.
The schema is simple enough that it could also suits the way Pyramid works.
Basically we could have wsgi workers that would consume an ASGI queue filled in with Daphnee
The main reason why I'd like pyramid to embrace this model is that it let you trigger asynchronous tasks in the request/response flow (send emails, process push notifications etc)
Do you have some thoughts? Are there people interested to work with me on that?
Wouldn't http://aiopyramid.readthedocs.io/index.html project solve this already?
I also created https://github.com/AppEnlight/channelstream/tree/master/channelstream - project based on pyramid+gevent to provide channel based websocket broadcast server.
No that's not the same approach because aiopyramid is just a way to run a worker that calls pyramid route in an asyncio loop so that the worker can handle more requests. It requires you to start writting async code in your server and will work only with Python 3.
The proposition behind ASGI is much more about having event handlers, it doesn't require you to write async code, it is still compatible with Python 2.7 and it let you scale workers with a shared event stream rather than having an event loop per Python worker.
Nevertheless thank you for pointing aiopyramid, it looks like a great project to look at.
@Natim I have an interested in working on this. Have you started a branch already?
I didn't but if you do I would be glad to review your PR and help with it.
Basically we could have wsgi workers that would consume an ASGI queue filled in with Daphnee
Taking ASGI requests and turning them into WSGI wouldn't require any interaction with Pyramid. Pyramid wouldn't even need to be aware of the fact that this was ASGI on the other end.
This is actually explicitly mentioned in the channels docs for the ASGI protocol:
http://channels.readthedocs.io/en/latest/asgi.html#wsgi-compatibility
Make this a generic WSGI server that instead of listening for HTTP (like waitress) instead listens on an ASGI message bus, and turns the http.request into a WSGI request, and routes the responses back onto the ASGI message bus.
If you wanted the ability to write from Pyramid to ASGI (for example for queuing something to be done later, like Celery) you could probably write a shim that allowed you to connect and send messages to ASGI just like you would currently connect to SQLAlchemy, Redis or Celery.
For the web worker part, we probably just need to convert ASGI calls to Daphnee into WSGI calls as explained by @bertjwregeer.
Then we might want to add helpers to handle other kind of ASGI message passing with Pyramid (Websocket handlers, tasks handlers) the shim @bertjwregeer is talking about.
@Natim All of the things I mentioned can be done outside of pyramid. The ASGI to WSGI would be useful for all WSGI projects, not just Pyramid. The ASGI shim helpers for example could be a pyramid_asgi, which also dealt with the different backends that may be used for message passing (Daphne uses Redis IIRC).
None of the work needs to be done in Pyramid core.
I don't see any reason this work can't be done as a normal addon to Pyramid. Can I close this issue, or is there a reason you think that this support is impossible if not done in core? I'm having difficulty deciding if this issue is here to gauge interest, or if it's calling for actual changes to Pyramid.
There is no reason this could not be done as an addon to Pyramid.
I don't know what is your policy around issues but it is mainly there as a starting point for anyone that would like to do the same thing. Feel free to close it or leave it open, I wish that people working around this subject can subscribe to it in order to aggregate the information around it.
I have no problem leaving it open if it is helpful for people. It had stagnated for a couple months and so I was interested in closing it if it had no effect on core and no one was using it.
This thread is one of the top search result for "websocket pyramid" and definitely the only one which contains the most up-to-date information. If someone makes a plugin or has some updates it'd be great to post it here.
I've made a websocket server based on gevent and pyramid - can be used standalone (as a broadcast server for any application - thats how i use it in production) or serve as an example on how to integrate things.
Reference: https://github.com/AppEnlight/channelstream
As far as updating Pyramid's support for ASGI I think by far the most useful thing that could be done right now is actually implementing and documenting how to write a Pyramid app that uses ASGI. We'd be happy to add it to the cookbook. Depending on how that went we would then talk about adding docs recommendations or features to Pyramid itself to point people in the right direction.
No we are slowly moving to Rust and tokio nowadays
I have created an example ASGI app for Pyramid that may be viewed here: https://github.com/erm/asgi-examples/. It is uses an adapter class to convert the WSGI requests to ASGI responses.
To keep this moving forward, instructions for setting up and getting it running could be added to the Deployment section in the Pyramid Cookbook. Here's its repo.
That's so nice! In theory, do you think Pyramid's SQLAlchemy template can be ported to ASGI + GINO (asyncpg + SQLAlchemy)?
@hyperknot I'm not too familiar with how Pyramid uses SQLAlchemy, but if presented as an ASGI application it may be possible.
Thanks to @ERM in Pylons/pyramid_cookbook#198 we now have a Pyramid Community Cookbook recipe for ASGI in Pyramid.
Per #2603 (comment) and to keep this issue moving forward:
- Are there features that should be added to Pyramid to support ASGI?
- What needs to be documented, including references to third parties?
- Where would it go in Pyramid docs?
To help keep moving this along (assuming there is still interest):
I am interested in assisting with ASGI support here. There have been some significant ASGI spec discussions taking place here that I am awaiting the outcome and can dig deeper into what needs to be considered for Pyramid support at that time.
Also, I've compiled a lot of the existing information regarding the ASGI ecosystem in a repo here to serve as a reference.
A couple more references to Flask 2.0 and its new support for async.
There is something wrong with the extended recipe, I get ValueError: WSGI wrapper received a non-HTTP scope
@ztane do you mean this recipe?
https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/deployment/asgi.html
That's in a separate repo at:
https://github.com/Pylons/pyramid_cookbook
It's possible that Pyramid 2.0 or another dependency broke the recipe, and it needs to be updated. Would you be willing and able to make an update, if necessary?