#Redis Crash Tester for Heroku Redis This small app is to demonstrate that Redis provided by Heroku can crash when multiple tasks are run against it. We started encountering the problem when we used Heroku Redis as a backend for Job-Queueing system and thus this sample program emulates that same scenario.
The repo contains two binaries that can be created. The first program is the worker which looks for new jobs in the queue (which is held in the Redis server). The second program is an enqueuer program which puts a specified number of jobs into the redis system at a very fast rate (using go routines).
This app was compiled and tested against go 1.15.7 but should produce the same output with go 1.14.x and 1.13.x as well.
It uses the go modules feature.
- Clone the repo using
git clone
. We are going to assume that the repo is being cloned into~/go/src/redis-crash
- Launch terminal and
cd
to the cloned directory (cd ~/go/src/redis-crash
) - Run the following command to compile and run the worker:
go build -o tmp/worker ./cmd/worker && REDIS_BG_URL=redis://127.0.0.1 tmp/worker
. This will compile the worker and run it. The worker will start looking for jobs in the redis installation pointed to byREDIS_BG_URL
- Run the following command to compile and run the enqueuer:
go build -o tmp/testJobEnqueuer ./cmd/testJobEnqueuer && JOB_COUNT=1000 tmp/testJobEnqueuer
. This will compile the enqueuer and launch it to enqueue 1000 jobs (governed byJOB_COUNT
variable) in the redis instance pointed to by theREDIS_BG_URL
.
Change the REDIS_BG_URL
value to test different Redis installations.
NOTE: Please make sure that both the worker and the enqueuer are using the same Redis installation.
You should run one worker (to consume) and 5+ enqueuers at the same time (don't just increase the job count, increase the number of simultaneous enqueuers).
NOTE 2: Either that or you can increase the number of connections in the pool near the max connections limit and run it.
In fact, the worker is optional; the only purpose it would serve for this demo is to drain the jobs so that Redis server does not fill up. We are not encountering any issues with the worker. It's the enqueueing that fails.
The error messages that we see (mostly on the 3rd, 4th, 5th terminal run of the enqueuer) are like these:
E#7X6OQ - Could not enqueue: EOF
E#7X6OQ - Could not enqueue: read tcp 192.168.29.50:56565->52.22.235.152:9390: read: connection reset by peer
In your own run, the source IP address (192.168.29.50
) should be something different.
This never happened with any other Redis installation we tested. SSL/TLS connection does not seem to be the issue, neither does it look like it is because of the client connections count (it never crossed 16 from what I can see in the Heroku Dashboard).
Heroku dashboard never shows us hitting the connection limit though!