client dashboard 404

Question

client dashboard 404

msmicker opened this issue 6 years ago · 26 comments

With 2.15.0 of dask and dask distributed I now get 404 errors in the client dashboard. When downgrading, everything works fine. I am using a cluster.

Answer 1 · 2020-04-26T22:18:20.000Z

Thank you for reporting. Were there any log messages that came out of the scheduler process?

Answer 2 · 2020-04-26T22:18:36.000Z

Those will be helpful to determine what went wrong.

Answer 3 · 2020-04-28T12:50:04.000Z

I'm going to close this issue for now as I suspect the 404 errors were related to #3746. @msmicker feel free to re-open if you're still experiencing issues

Answer 4 · 2020-04-29T02:13:10.000Z

I still see this error with 2.15.1. @mrocklin apologies but how do I check distributed logs? I am not yet scheduling anything. Simply starting the cluster and client and then opening the dashboard.

from dask_jobqueue import SGECluster
from dask.distributed import Client

cluster = SGECluster(...)

cluster.adapt(minimum_jobs=5, maximum_jobs=150)

client = Client(cluster)
client

client.get_scheduler_logs()

(('INFO', 'distributed.scheduler - INFO - Clear task state'),
 ('INFO',
  'distributed.scheduler - INFO -   Scheduler at: tcp://157.206.230.212:38448'),
 ('INFO',
  'distributed.scheduler - INFO -   dashboard at:                     :8787'),
 ('INFO',
  'distributed.scheduler - INFO - Receive client connection: Client-cc91b07d-89c3-11ea-9dc8-94f128bfa670'),
 ('INFO',
  "distributed.scheduler - INFO - Register worker <Worker 'tcp://157.206.230.55:42049', name: 3, memory: 0, processing: 0>"),
 ('INFO',
  'distributed.scheduler - INFO - Starting worker compute stream, tcp://157.206.230.55:42049'),
 ('INFO',
  "distributed.scheduler - INFO - Register worker <Worker 'tcp://157.206.230.76:34078', name: 0, memory: 0, processing: 0>"),
 ('INFO',
  'distributed.scheduler - INFO - Starting worker compute stream, tcp://157.206.230.76:34078'),
 ('INFO',
  "distributed.scheduler - INFO - Register worker <Worker 'tcp://157.206.230.76:33739', name: 1, memory: 0, processing: 0>"),
 ('INFO',
  'distributed.scheduler - INFO - Starting worker compute stream, tcp://157.206.230.76:33739'),
 ('INFO',
  "distributed.scheduler - INFO - Register worker <Worker 'tcp://157.206.230.76:32793', name: 2, memory: 0, processing: 0>"),
 ('INFO',
  'distributed.scheduler - INFO - Starting worker compute stream, tcp://157.206.230.76:32793'),
 ('INFO',
  "distributed.scheduler - INFO - Register worker <Worker 'tcp://157.206.230.55:34488', name: 4, memory: 0, processing: 0>"),
 ('INFO',
  'distributed.scheduler - INFO - Starting worker compute stream, tcp://157.206.230.55:34488'))

client.get_worker_logs()

{'tcp://157.206.230.55:34488': (('INFO',
   'distributed.worker - INFO -       Start worker at: tcp://157.206.230.55:34488'),
  ('INFO',
   'distributed.worker - INFO -          Listening to: tcp://157.206.230.55:34488'),
  ('INFO',
   'distributed.worker - INFO -          dashboard at:       157.206.230.55:34286'),
  ('INFO',
   'distributed.worker - INFO - Waiting to connect to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -               Threads:                          1'),
  ('INFO',
   'distributed.worker - INFO -                Memory:                   24.00 GB'),
  ('INFO',
   'distributed.worker - INFO -       Local Directory: '),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -         Registered to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------')),
 'tcp://157.206.230.55:42049': (('INFO',
   'distributed.worker - INFO -       Start worker at: tcp://157.206.230.55:42049'),
  ('INFO',
   'distributed.worker - INFO -          Listening to: tcp://157.206.230.55:42049'),
  ('INFO',
   'distributed.worker - INFO -          dashboard at:       157.206.230.55:39319'),
  ('INFO',
   'distributed.worker - INFO - Waiting to connect to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -               Threads:                          1'),
  ('INFO',
   'distributed.worker - INFO -                Memory:                   24.00 GB'),
  ('INFO',
   'distributed.worker - INFO -       Local Directory: '),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -         Registered to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------')),
 'tcp://157.206.230.76:32793': (('INFO',
   'distributed.worker - INFO -       Start worker at: tcp://157.206.230.76:32793'),
  ('INFO',
   'distributed.worker - INFO -          Listening to: tcp://157.206.230.76:32793'),
  ('INFO',
   'distributed.worker - INFO -          dashboard at:       157.206.230.76:34934'),
  ('INFO',
   'distributed.worker - INFO - Waiting to connect to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -               Threads:                          1'),
  ('INFO',
   'distributed.worker - INFO -                Memory:                   24.00 GB'),
  ('INFO',
   'distributed.worker - INFO -       Local Directory: '),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -         Registered to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------')),
 'tcp://157.206.230.76:33739': (('INFO',
   'distributed.worker - INFO -       Start worker at: tcp://157.206.230.76:33739'),
  ('INFO',
   'distributed.worker - INFO -          Listening to: tcp://157.206.230.76:33739'),
  ('INFO',
   'distributed.worker - INFO -          dashboard at:       157.206.230.76:37945'),
  ('INFO',
   'distributed.worker - INFO - Waiting to connect to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -               Threads:                          1'),
  ('INFO',
   'distributed.worker - INFO -                Memory:                   24.00 GB'),
  ('INFO',
   'distributed.worker - INFO -       Local Directory: '),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -         Registered to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------')),
 'tcp://157.206.230.76:34078': (('INFO',
   'distributed.worker - INFO -       Start worker at: tcp://157.206.230.76:34078'),
  ('INFO',
   'distributed.worker - INFO -          Listening to: tcp://157.206.230.76:34078'),
  ('INFO',
   'distributed.worker - INFO -          dashboard at:       157.206.230.76:38661'),
  ('INFO',
   'distributed.worker - INFO - Waiting to connect to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -               Threads:                          1'),
  ('INFO',
   'distributed.worker - INFO -                Memory:                   24.00 GB'),
  ('INFO',
   'distributed.worker - INFO -       Local Directory: '),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   'distributed.worker - INFO -         Registered to: tcp://157.206.230.212:38448'),
  ('INFO',
   'distributed.worker - INFO - -------------------------------------------------'))}

Answer 5 · 2020-04-29T03:18:03.000Z

With SGECluster there may be a silence_logs=False keyword option. I don't remember exactly though.

Answer 6 · 2020-05-06T03:45:50.000Z

@mrocklin just to check if logs I added via update were enough or should I try for more (e.g. with silence_logs=True)

Answer 7 · 2020-05-07T11:22:32.000Z

I have the same problem. I'm using dask-jobqueue with a HTCondor Cluster.

Answer 8 · 2020-05-07T11:24:58.000Z

@msmicker @lenaWitterauf can you try with 2.15.2? It includes a second fix that others have reported success with.

Answer 9 · 2020-05-07T12:41:52.000Z

@TomAugspurger I just updated dask distributed to 2.15.2 and the dashboard worked! Thanks a lot :-)

Answer 10 · 2020-05-08T20:25:22.000Z

Same, all is good with 2.15.2, thanks!

Answer 11 · 2020-05-28T14:33:19.000Z

For anyone stumbling upon this issue by searching for dask dashboard 404:

You will see 404 if you try to run it in an environment without bokeh installed.

Answer 12 · 2020-05-28T14:38:07.000Z

Thanks for engaging @elanmart. There's an open PR (xref dask/dask#6215) to add a note to the docs about getting 404 errors when bokeh isn't installed. Hopefully that will help others running into this issue

Answer 13 · 2020-05-28T14:41:47.000Z

@jrbourbeau oops, in that case apologies for adding noise, thanks for the reference!

Answer 14 · 2020-05-28T14:46:08.000Z

Perhaps we should include a vanilla route under `/status` (the common entry point for Bokeh users) that says "please install bokeh" with some conda/pip install lines if Bokeh isn't installed. The relevant code is probably in distributed/http/scheduler/info.py . I suspect that someone even very mildly familiar with web programming could accomplish this by copying examples that are already there.

…

On Thu, May 28, 2020 at 7:33 AM Marcin Elantkowski ***@***.***> wrote: For anyone stumbling upon this issue by searching for dask dashboard 404: You will see 404 if you try to run it in an environment without bokeh installed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3741 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTFPC2ZHZTQAIUKN6LDRTZY35ANCNFSM4MRNXIHA> .

Answer 15 · 2020-05-28T15:01:16.000Z

@dankerrigan you might find this interesting

Answer 16 · 2020-05-28T15:06:15.000Z

That seems like a nice improvement @mrocklin. I've opened up a separate issue (xref #3836) for tracking

Answer 17 · 2020-06-15T17:23:33.000Z

For anyone stumbling upon this issue by searching for dask dashboard 404:

You will see 404 if you try to run it in an environment without bokeh installed.

I have bokeh installed but unfortunately I'm still getting 404: Not Found when visiting dashboard address.

Answer 18 · 2020-06-16T03:45:41.000Z

Same here:

$ /deploy/env/bin/pip freeze | grep bokeh
bokeh==2.1.0
$ curl localhost:8787
<html><title>404: Not Found</title><body>404: Not Found</body></html>

Answer 19 · 2020-06-16T03:56:56.000Z

If it helps, this reliably reproduces it for me:

$ virtualenv -p python3 dask
$ source dask/bin/activate
$ pip install 'dask[distributed,diagnostics]'
$ pip freeze | grep bokeh
bokeh==2.1.0
$ pip freeze | grep dask
dask==2.18.1
$ dask-scheduler
# localhost:8787 now 404s

Answer 20 · 2020-06-16T13:31:17.000Z

Yes, reproduced. It looks like an internal function in bokeh moved in bokeh 2.1.0. I'll resolve this on our end. In the meantime, I recommend downgrading to bokeh 2.0.

In the future, I also recommend raising a new issue rather than commenting on an old one, although the symptoms are the same as this old issue, the cause is entirely new.

Answer 21 · 2020-06-16T13:33:05.000Z

Thanks! Apologies, the issue seemed pretty fresh.

Answer 22 · 2020-06-16T13:33:43.000Z

No need to apologize, and thank you for providing explicit steps to reproduce. This is a serious issue and I thank you for catching and reporting it.

Answer 23 · 2020-06-16T13:44:59.000Z

OK, the critical fix is in here: #3904

Additionally, it looks like there is a deserialiation warning here: #3905

Answer 24 · 2020-10-09T02:56:22.000Z

I'm seeing this as well, and I've got bokeh installed (distributed 2.18.0).

Also shouldn't this issue be reopened?

Answer 25 · 2020-10-09T03:40:58.000Z

@CMCDragonkai I'd recommend opening a new issue with a minimal reproducer

Answer 26 · 2020-11-20T16:48:19.000Z

Nevermind, upgrading distributed to 2.30 works!

I am also getting this:

from dask.distributed import Client

client = Client()
client