AWS subscriptions don't always publish
Closed this issue · 13 comments
I'm having a weird bug happen on AWS where a subscription only publishes every other time that a certain mutation is performed. It does not do this when I run the server locally on localhost.
One thing I realized that may be causing this is that I'm using the in memory AsyncPubSub. It's possible that the server may be executing calls on 2 different instances. This may mean that the client is subscribing to one instance's subscriptions, but that the load balancer causes the mutations to switch off between different instances, so only half the time would the server with the open websocket be receiving the mutation and publishing it.
I will try to come up with a solution using Redis and the new PubSub protocol, but I wanted to get someone else's thoughts on this.
What type of AWS services are you using?
I am not too familiar with AWS and all of their types of services but I'll look into it
AWS ECS. It's a container service and I'm using Docker. I think it has 2 EC2 instances running, so my explanation might make sense.
I'll work on creating a RedisPubSub package that implements the Pioneer PubSub and hopefully that will fix it.
Right, it's likely the case here.
Do keep me updated if the issue still exist even with a Redis backed PubSub.
I'll work on creating a RedisPubSub package that implements the Pioneer PubSub and hopefully that will fix it.
Also, you can use the example I put out in the documentation as a starting point in building the Redis backed PubSub.
Oh wow lol. That's pretty much exactly what I needed. You should really just put that in a new repo called PioneerRedisPubSub. I was going to do it, but you've already done all the work. You might as well get the credit.
Oh wow lol. That's pretty much exactly what I needed. You should really just put that in a new repo called
PioneerRedisPubSub. I was going to do it, but you've already done all the work. You might as well get the credit.
I will probably put it in a new repo, but I have only tested it in an example application once so I will be making sure that it fully working and clean of major issues beforehand.
I couldn't find anything on AsyncPubSub behaving differently running on a docker container locally and hosted on a VM machine (setting the environment to production and development change nothing as well). All published events are received properly. So unless AWS ECS handles Docker containers differently, I think it was running on 2 or more instances in your case.
I have made the repo named PioneerRedisPubSub. Although, I have to admit I spend an embarrassingly long time creating a working RedisClient with proper configurations that work and behave properly during testing, and I am still not sure whether that client is even correctly set up.
Haha that was fast! I think it looks really good. The redis configuration specifics really shouldn't matter that much for the package right? It's basically an abstraction on top of Redis PubSub that lets the developer use it like AsyncPubSub.
Yeah, hopefully
A bit of update here, I managed to get proper working unit tests for the PioneerRedisPubSub library. It should be behaving more consistently now.
This issue has been open for a while. I have not seen any updates and I am assuming that the issue is resolved for now.
Feel free to open this issue again if the issue still exist