Upgrading to redis>=3.0 results in failed job

[remesta]

With redis==2.10.6 all is working as expected. After upgrading to the latest redis>=3.0 jobs start to fail with AttributeError: 'int' object has no attribute 'items' exceptions.

---

I created a simple job:

from flask_rq2 import RQ

rq = RQ()

@rq.job
def add(x, y):
    return x + y

Running Flask RQ worker as usual: $ flask rq worker --worker-ttl=420

08:51:13 RQ worker 'rq:worker:localhost.4064' started, version 0.12.0
08:51:13 *** Listening on default...
08:51:13 Cleaning registries for queue: default
08:51:22 default: flask_demo.jobs.add(1, 2) (48624d9a-44fd-4ce5-8190-35d1d1a44a8b)
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/flask_demo/bin/flask", line 11, in <module>
    sys.exit(main())
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/flask/cli.py", line 894, in main
    cli.main(args=args, prog_name=name)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/flask/cli.py", line 557, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/flask/cli.py", line 412, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/flask_rq2/cli.py", line 55, in new_func
    return func(rq, ctx, *args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/flask_rq2/cli.py", line 154, in worker
    **shared_options(rq)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/cli/cli.py", line 75, in wrapper
    return ctx.invoke(func, cli_config, *args[1:], **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/cli/cli.py", line 236, in worker
    worker.work(burst=burst, logging_level=logging_level)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/worker.py", line 493, in work
    self.execute_job(job, queue)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/worker.py", line 662, in execute_job
    self.fork_work_horse(job, queue)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/worker.py", line 599, in fork_work_horse
    self.main_work_horse(job, queue)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/worker.py", line 677, in main_work_horse
    success = self.perform_job(job, queue)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/worker.py", line 781, in perform_job
    self.prepare_job_execution(job)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/worker.py", line 706, in prepare_job_execution
    registry.add(job, timeout, pipeline=pipeline)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/rq/registry.py", line 47, in add
    return pipeline.zadd(self.key, score, job.id)
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/redis/client.py", line 2263, in zadd
    for pair in iteritems(mapping):
  File "/Users/user/.virtualenvs/flask_demo/lib/python3.7/site-packages/redis/_compat.py", line 123, in iteritems
    return iter(x.items())
AttributeError: 'int' object has no attribute 'items'
08:51:22 Moving job to 'failed' queue (work-horse terminated unexpectedly; waitpid returned 256)

[tuangeek]

I ran into this too. rq has an open issue for it rq/rq#1014 They suggest using the following until it is fixed.

redis==2.10.6
rq==0.12.0

[sbehrens]

Also ran into the issue, pinning to redis 2.10.6 resolved it.

[jezdez]

I've released 18.2 that requires RQ < 0.13.0 and redis < 3.0.0 while we wait for the fix in RQ.

[selwin]

rq/rq#1016 has been merged in. I'll wait for a few days before cutting a release, just to be doubly sure that there are no major bugs introduced.

[jezdez]

@selwin Thank you!

[jezdez]

There is more to do in rq/rq-scheduler#197.

[jezdez]

I've published a 18.2.2 release that should fix the version conflict as reported in #75. There is also a 18.3 release that requires redis-py 3.0, rq 0.13 and rq-scheduler 0.9 and fixes this ticket for good. Please use 18.2.2 if you need to stay on redis-py < 3.0.

[janzjos]

Using this thread, i've used various combinations of pinning redis, rq, and Flask-RQ2 and still get the error

Moving job to 'failed' queue (work-horse terminated unexpectedly; waitpid returned

I have also made sure my timeout in seconds for the @rq.job decorator is set. Is there a configuration i'm missing here to ensure large jobs are not sent to failed?