"No diskover indices found in Elasticsearch" when testing via Docker

[axfelix]

After running $ docker-compose build && docker-compose up per the readme, and navigating to localhost:8080, the app has a message that says No diskover indices found in Elasticsearch. Please run a crawl and come back. and needs me to specify Index and Index 2 before I can proceed. Elasticsearch and Redis appear to have been deployed successfully via Docker; do I need to POST to Elasticsearch directly at localhost:9200 to get it to do something, or should I be editing config inside the Docker container? It's not clear from the documentation.

[axfelix]

Doing a simple curl -X PUT "localhost:9200/diskover" is sufficient to get diskover-web to recognize the index and allow me to select a value and click through to the next page, but it throws a 400, presumably because the index is totally empty. Am I expected to clone and run diskover itself after using docker to install discover-web and the ES and Redis dependencies?

I tried doing that by running $ python diskover_worker_bot.py and then running an example command like python diskover.py -d /home/axfelix/Dropbox/Documents -i diskover-test -a -O but that caused the worker bot to die like this:

Traceback (most recent call last):
  File "diskover_worker_bot.py", line 47, in <module>
    w.work()
  File "/usr/lib/python3.7/site-packages/rq/worker.py", line 493, in work
    self.execute_job(job, queue)
  File "/usr/lib/python3.7/site-packages/rq/worker.py", line 911, in execute_job
    return self.perform_job(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/rq/worker.py", line 781, in perform_job
    self.prepare_job_execution(job)
  File "/usr/lib/python3.7/site-packages/rq/worker.py", line 706, in prepare_job_execution
    registry.add(job, timeout, pipeline=pipeline)
  File "/usr/lib/python3.7/site-packages/rq/registry.py", line 47, in add
    return pipeline.zadd(self.key, score, job.id)
  File "/usr/lib/python3.7/site-packages/redis/client.py", line 2263, in zadd
    for pair in iteritems(mapping):
  File "/usr/lib/python3.7/site-packages/redis/_compat.py", line 123, in iteritems
    return iter(x.items())
AttributeError: 'int' object has no attribute 'items'

I guess that might be an issue with system Python being 3.7 as you aren't supporting it yet? But it's a little awkward to deploy everything else via Docker and then encounter this issue running diskover itself when installed manually...

[axfelix]

Seems to be the same for Python 3.6, and I can't seem to wipe the queue now either -- whenever I start a new worker it dies on the same queued task the same way.

[axfelix]

Was able to clear that with $ redis-cli flushall but still getting the AttributeError whenever I try to index anything. This might be an issue with diskover and not diskover-web at this point unless I've seriously misunderstood deployment.

[shirosaidev]

Please make sure you have the versions of python, redis, rq, elasticsearch in the readme of diskover. You could also try with the linuxserver.io docker hub image which might make it easier to get it running.

[YangZhou123]

Same issue!
Traceback (most recent call last):
File "diskover_worker_bot.py", line 47, in
w.work()
File "/home/vagrant/.local/lib/python3.5/site-packages/rq/worker.py", line 493, in work
self.execute_job(job, queue)
File "/home/vagrant/.local/lib/python3.5/site-packages/rq/worker.py", line 911, in execute_job
return self.perform_job(*args, **kwargs)
File "/home/vagrant/.local/lib/python3.5/site-packages/rq/worker.py", line 781, in perform_job
self.prepare_job_execution(job)
File "/home/vagrant/.local/lib/python3.5/site-packages/rq/worker.py", line 706, in prepare_job_execution
registry.add(job, timeout, pipeline=pipeline)
File "/home/vagrant/.local/lib/python3.5/site-packages/rq/registry.py", line 47, in add
return pipeline.zadd(self.key, score, job.id)
File "/home/vagrant/.local/lib/python3.5/site-packages/redis/client.py", line 2263, in zadd
for pair in iteritems(mapping):
File "/home/vagrant/.local/lib/python3.5/site-packages/redis/_compat.py", line 123, in iteritems
return iter(x.items())
AttributeError: 'int' object has no attribute 'items'

python version 3.5.2

[shirosaidev]

which version of rq and redis python modules are installed?

[shirosaidev]

seems like this is a bug with redis python module 3.0.1 (latest) and latest rq 0.12.0, please use redis python module 2.10.6.

[shirosaidev]

pip3 install "redis>=2.0.0,<3.0.0"

[YangZhou123]

Thanks, Chris. It works now!