Can't access TLJH via a browser
ParkvilleData opened this issue · 17 comments
Hi,
I have tried installing TLJH via these commands (https://the-littlest-jupyterhub.readthedocs.io/en/latest/install/custom-server.html)
Installation works correctly and I get the "Done!" output.
But, when I try to access http://<global_ip_address> (with an actual IP address) I get a page not found error. I am running apache and I know that I can access the IP address externally from the machine.
Http access is enabled. SSH is enabled. It just doesn't seem to be running. Any help would be greatly appreciated.
Thanks,
Hi @ParkvilleData! Thanks for opening the issue. Can you please share the traefik and jupyterhub logs?
E.g.:
sudo journalctl -u traefik
sudo journalctl -u jupyterhub
Maybe there's something there that may help us see what's going on.
Hi GeorgianaElena, thanks for the response. Here is the output.
$ sudo journalctl -u traefik
-- Logs begin at Mon 2019-09-23 01:02:44 UTC, end at Mon 2019-09-23 21:47:54 UTC. --
Sep 23 01:30:06 jupyterhub systemd[1]: Started traefik.service.
Sep 23 01:30:15 jupyterhub traefik[5544]: time="2019-09-23T01:30:07Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 23 01:30:15 jupyterhub traefik[5544]: time="2019-09-23T01:30:07Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 23 01:30:15 jupyterhub traefik[5544]: time="2019-09-23T01:30:07Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 23 01:30:15 jupyterhub traefik[5544]: time="2019-09-23T01:30:07Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> Forw
Sep 23 01:30:15 jupyterhub traefik[5544]: time="2019-09-23T01:30:07Z" level=fatal msg="Error preparing server: error opening listener: listen tcp :80: bind: address already in use"
Sep 23 01:30:15 jupyterhub traefik[5603]: time="2019-09-23T01:30:07Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 23 01:30:15 jupyterhub traefik[5603]: time="2019-09-23T01:30:07Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 23 01:30:15 jupyterhub traefik[5603]: time="2019-09-23T01:30:07Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 23 01:30:15 jupyterhub traefik[5603]: time="2019-09-23T01:30:07Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> Forw
Sep 23 01:30:15 jupyterhub traefik[5603]: time="2019-09-23T01:30:07Z" level=fatal msg="Error preparing server: error opening listener: listen tcp :80: bind: address already in use"
Sep 23 01:30:07 jupyterhub systemd[1]: Stopping traefik.service...
Sep 23 01:30:07 jupyterhub systemd[1]: Stopped traefik.service.
Sep 23 01:30:15 jupyterhub traefik[5619]: time="2019-09-23T01:30:08Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 23 01:30:15 jupyterhub traefik[5619]: time="2019-09-23T01:30:08Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 23 01:30:15 jupyterhub traefik[5619]: time="2019-09-23T01:30:08Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 23 01:30:15 jupyterhub traefik[5619]: time="2019-09-23T01:30:08Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> Forw
Sep 23 01:30:15 jupyterhub traefik[5619]: time="2019-09-23T01:30:08Z" level=fatal msg="Error preparing server: error opening listener: listen tcp :80: bind: address already in use"
Sep 23 01:30:07 jupyterhub systemd[1]: Started traefik.service.
Sep 23 01:30:15 jupyterhub traefik[5635]: time="2019-09-23T01:30:08Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 23 01:30:15 jupyterhub traefik[5635]: time="2019-09-23T01:30:08Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 23 01:30:15 jupyterhub traefik[5635]: time="2019-09-23T01:30:08Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 23 01:30:15 jupyterhub traefik[5635]: time="2019-09-23T01:30:08Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS:<nil> Redirect:<nil> Auth:0xc0002d5bc0 WhitelistSourceRange:[] WhiteList:0xc00033ac60 Compress:f
Sep 23 01:30:15 jupyterhub traefik[5635]: time="2019-09-23T01:30:08Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> Forw
Sep 23 01:30:15 jupyterhub traefik[5635]: time="2019-09-23T01:30:08Z" level=fatal msg="Error preparing server: error opening listener: listen tcp :80: bind: address already in use"
Sep 23 01:30:15 jupyterhub traefik[5652]: time="2019-09-23T01:30:09Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 23 01:30:15 jupyterhub traefik[5652]: time="2019-09-23T01:30:09Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 23 01:30:15 jupyterhub traefik[5652]: time="2019-09-23T01:30:09Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 23 01:30:15 jupyterhub traefik[5652]: time="2019-09-23T01:30:09Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> Forw
Sep 23 01:30:15 jupyterhub traefik[5652]: time="2019-09-23T01:30:09Z" level=fatal msg="Error preparing server: error opening listener: listen tcp :80: bind: address already in use"
Sep 23 01:30:07 jupyterhub systemd[1]: traefik.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 01:30:07 jupyterhub systemd[1]: traefik.service: Failed with result 'exit-code'.
Sep 23 01:30:15 jupyterhub traefik[5668]: time="2019-09-23T01:30:09Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 23 01:30:15 jupyterhub traefik[5668]: time="2019-09-23T01:30:09Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 23 01:30:15 jupyterhub traefik[5668]: time="2019-09-23T01:30:09Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 23 01:30:15 jupyterhub traefik[5668]: time="2019-09-23T01:30:09Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> Forw
Sep 23 01:30:15 jupyterhub traefik[5668]: time="2019-09-23T01:30:09Z" level=fatal msg="Error preparing server: error opening listener: listen tcp :80: bind: address already in use"
Sep 23 01:30:07 jupyterhub systemd[1]: traefik.service: Service hold-off time over, scheduling restart.
Sep 23 01:30:07 jupyterhub systemd[1]: traefik.service: Scheduled restart job, restart counter is at 1.
Sep 23 01:30:07 jupyterhub systemd[1]: Stopped traefik.service.
Sep 23 01:30:07 jupyterhub systemd[1]: Started traefik.service.
Sep 23 01:30:07 jupyterhub systemd[1]: traefik.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 01:30:07 jupyterhub systemd[1]: traefik.service: Failed with result 'exit-code'.
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Service hold-off time over, scheduling restart.
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Scheduled restart job, restart counter is at 2.
Sep 23 01:30:08 jupyterhub systemd[1]: Stopped traefik.service.
Sep 23 01:30:08 jupyterhub systemd[1]: Started traefik.service.
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Failed with result 'exit-code'.
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Service hold-off time over, scheduling restart.
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Scheduled restart job, restart counter is at 3.
Sep 23 01:30:08 jupyterhub systemd[1]: Stopped traefik.service.
Sep 23 01:30:08 jupyterhub systemd[1]: Started traefik.service.
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 01:30:08 jupyterhub systemd[1]: traefik.service: Failed with result 'exit-code'.
Sep 23 01:30:09 jupyterhub systemd[1]: traefik.service: Service hold-off time over, scheduling restart.
Sep 23 01:30:09 jupyterhub systemd[1]: traefik.service: Scheduled restart job, restart counter is at 4.
Sep 23 01:30:09 jupyterhub systemd[1]: Stopped traefik.service.
Sep 23 01:30:09 jupyterhub systemd[1]: Started traefik.service.
Sep 23 01:30:09 jupyterhub systemd[1]: traefik.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 01:30:09 jupyterhub systemd[1]: traefik.service: Failed with result 'exit-code'.
$ sudo journalctl -u jupyterhub
-- Logs begin at Mon 2019-09-23 01:02:44 UTC, end at Mon 2019-09-23 21:48:44 UTC. --
Sep 23 01:30:06 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 01:30:06 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 01:30:07 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 01:30:07 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 01:30:07 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 01:30:07 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 01:30:07 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 01:30:08 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 01:30:08 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 01:30:08 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 01:30:08 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 01:30:08 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 01:30:08 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 01:30:09 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 01:30:09 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 01:30:09 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 01:30:09 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 01:30:09 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 01:30:09 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 01:30:10 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 01:30:10 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 01:30:10 jupyterhub systemd[1]: Dependency failed for jupyterhub.service.
Sep 23 01:30:10 jupyterhub systemd[1]: jupyterhub.service: Job jupyterhub.service/start failed with result 'dependency'.
Sep 23 02:40:52 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 02:40:53 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 02:40:53 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 02:40:53 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 02:40:53 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 02:40:53 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 02:40:53 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 02:40:54 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 02:40:54 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 02:40:54 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 02:40:54 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 02:40:54 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 02:40:54 jupyterhub systemd[1]: Started jupyterhub.service.
Sep 23 02:40:55 jupyterhub systemd[1]: Stopping jupyterhub.service...
Sep 23 02:40:55 jupyterhub systemd[1]: Stopped jupyterhub.service.
Sep 23 02:40:55 jupyterhub systemd[1]: Dependency failed for jupyterhub.service.
Sep 23 02:40:55 jupyterhub systemd[1]: jupyterhub.service: Job jupyterhub.service/start failed with result 'dependency'.
Thanks @ParkvilleData! It looks like the proxy is not able to start on port 80 because the port is being used by another process. If freeing the port is not an option, you can change the proxy port using tljh-config. E.g.:
sudo tljh-config set http.port <new_proxy_port>
sudo tljh-config reload proxy
You should then be able to access the Hub in the browser at http://<global_ip_address>:new_proxy_port>.
Hope this helps.
Hi GeorgianaElena,
I tried that but it didn't work. I added port 100 for traefik as below. There is no difference to the Jupyterhub.service log.
sudo lsof -i -P -n | grep LISTEN.
"systemd-r 491 systemd-resolve 13u IPv4 13055 0t0 TCP 127.0.0.53:53 (LISTEN)
apache2 886 www-data 4u IPv6 162228 0t0 TCP *:80 (LISTEN)
apache2 1477 www-data 4u IPv6 162228 0t0 TCP *:80 (LISTEN)
sshd 2273 root 3u IPv4 25716 0t0 TCP *:22 (LISTEN)
sshd 2273 root 4u IPv6 25718 0t0 TCP *:22 (LISTEN)
traefik 6320 root 3u IPv6 233152 0t0 TCP *:100 (LISTEN)
traefik 6320 root 5u IPv4 233153 0t0 TCP 127.0.0.1:8099 (LISTEN)
apache2 32615 www-data 4u IPv6 162228 0t0 TCP *:80 (LISTEN)
apache2 32616 www-data 4u IPv6 162228 0t0 TCP *:80 (LISTEN)
apache2 32617 www-data 4u IPv6 162228 0t0 TCP *:80 (LISTEN)
apache2 32618 www-data 4u IPv6 162228 0t0 TCP *:80 (LISTEN)
apache2 32619 www-data 4u IPv6 162228 0t0 TCP *:80 (LISTEN)"
Traefik port is open and Listening.
"
Sep 25 21:50:33 jupyterhub systemd[1]: Started traefik.service.
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Preparing server http &{Address::100 TLS: Redirect: Auth: WhitelistSourceRange:[] WhiteList: Compress:false ProxyProtocol: For
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS: Redirect: Auth:0xc00004a930 WhitelistSourceRange:[] WhiteList:0xc0003a2a80 Compress:f
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting server on :100"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting provider *file.Provider {"Watch":true,"Filename":"rules.toml","Constraints":null,"Trace":false,"TemplateVersion":0,"DebugLogGener
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting server on 127.0.0.1:8099"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Server configuration reloaded on :100"
"
@ParkvilleData, try restarting the hub service with sudo tljh-config reload hub.
The hub service didn't start int the first place because it didn't find proxy service up and running and I think it might need a restart in order to re-check its dependencies (traefik service).
Thanks for the response. Unfortunately it still isn't working. The logs look much better but I am not seeing the page when using the addresss with the proxy port. The proxy port change to 100 has come into effect but still nothing.
I am able to create a simple html site and can view that externally using port 80/8080 but I can't see TLJH.
Any advice would be greatly appreciated, thank you!
Oct 03 01:05:40 jupyterhub systemd[1]: Started jupyterhub.service.
Oct 03 01:05:49 jupyterhub python3[1312]: [I 2019-10-03 01:05:49.621 JupyterHub app:2120] Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator
Oct 03 01:05:49 jupyterhub python3[1312]: [I 2019-10-03 01:05:49.622 JupyterHub app:2120] Using Spawner: builtins.UserCreatingSpawner
Oct 03 01:05:49 jupyterhub python3[1312]: [I 2019-10-03 01:05:49.735 JupyterHub app:1302] Writing cookie_secret to /opt/tljh/state/jupyterhub_cookie_secret
Oct 03 01:05:49 jupyterhub python3[1312]: [I 2019-10-03 01:05:49.946 alembic.runtime.migration migration:154] Context impl SQLiteImpl.
Oct 03 01:05:49 jupyterhub python3[1312]: [I 2019-10-03 01:05:49.953 alembic.runtime.migration migration:161] Will assume non-transactional DDL.
Oct 03 01:05:50 jupyterhub python3[1312]: [I 2019-10-03 01:05:50.034 alembic.runtime.migration migration:513] Running stamp_revision -> 4dc2d5a8c53c
Oct 03 01:05:50 jupyterhub python3[1312]: [I 2019-10-03 01:05:50.161 alembic.runtime.migration migration:154] Context impl SQLiteImpl.
Oct 03 01:05:50 jupyterhub python3[1312]: [I 2019-10-03 01:05:50.162 alembic.runtime.migration migration:161] Will assume non-transactional DDL.
and
Sep 25 21:50:33 jupyterhub systemd[1]: Started traefik.service.
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Preparing server http &{Address::100 TLS: Redirect: Auth: WhitelistSourceRange:[] WhiteList: Compress:false ProxyProtocol: For
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS: Redirect: Auth:0xc00004a930 WhitelistSourceRange:[] WhiteList:0xc0003a2a80 Compress:f
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting server on :100"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting provider *file.Provider {"Watch":true,"Filename":"rules.toml","Constraints":null,"Trace":false,"TemplateVersion":0,"DebugLogGener
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Starting server on 127.0.0.1:8099"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Sep 25 21:50:34 jupyterhub traefik[6320]: time="2019-09-25T21:50:34Z" level=info msg="Server configuration reloaded on :100"
Sep 25 22:00:37 jupyterhub traefik[6320]: time="2019-09-25T22:00:37Z" level=warning msg="A new release has been found: 1.7.17. Please consider updating."
Sep 26 21:50:37 jupyterhub traefik[6320]: time="2019-09-26T21:50:37Z" level=warning msg="A new release has been found: 2.0.1. Please consider updating."
Sep 27 21:50:37 jupyterhub traefik[6320]: time="2019-09-27T21:50:37Z" level=warning msg="A new release has been found: 2.0.1. Please consider updating."
Sep 28 21:50:37 jupyterhub traefik[6320]: time="2019-09-28T21:50:37Z" level=warning msg="A new release has been found: 2.0.1. Please consider updating."
Sep 29 21:50:37 jupyterhub traefik[6320]: time="2019-09-29T21:50:37Z" level=warning msg="A new release has been found: 2.0.1. Please consider updating."
Sep 30 21:50:37 jupyterhub traefik[6320]: time="2019-09-30T21:50:37Z" level=warning msg="A new release has been found: 2.0.1. Please consider updating."
Oct 01 21:50:37 jupyterhub traefik[6320]: time="2019-10-01T21:50:37Z" level=warning msg="A new release has been found: 2.0.1. Please consider updating."
Oct 02 21:50:37 jupyterhub traefik[6320]: time="2019-10-02T21:50:37Z" level=warning msg="A new release has been found: 2.0.1. Please consider updating."
Oct 03 01:05:51 jupyterhub traefik[6320]: time="2019-10-03T01:05:51Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Oct 03 01:05:51 jupyterhub traefik[6320]: time="2019-10-03T01:05:51Z" level=info msg="Server configuration reloaded on :100"
Hi,
Is there any update to this?
Thanks
Hello
I am seeing a similar issue. It looks like the traefik service is not coming up as port 80 is in use. This failure started happening after I ran apt update. What is the sequence for the services to start? Does traefik need to start before lighthttpd? This is the logs from journalctl -u traefik
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: Started traefik.service.
Feb 18 14:28:38 ip-10-0-2-138 traefik[2935]: time="2020-02-18T14:28:38Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Feb 18 14:28:38 ip-10-0-2-138 traefik[2935]: time="2020-02-18T14:28:38Z" level=info msg="Traefik version v1.7.18 built on 2019-09-26_01:56:30PM"
Feb 18 14:28:38 ip-10-0-2-138 traefik[2935]: time="2020-02-18T14:28:38Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v1.7/basics/#collected-data\n"
Feb 18 14:28:38 ip-10-0-2-138 traefik[2935]: time="2020-02-18T14:28:38Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0002fc020} with readTimeout=0s writeTimeout=0s idleTimeout=10m0s"
Feb 18 14:28:38 ip-10-0-2-138 traefik[2935]: time="2020-02-18T14:28:38Z" level=fatal msg="Error preparing server: error opening listener: listen tcp :80: bind: address already in use"
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: traefik.service: Main process exited, code=exited, status=1/FAILURE
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: traefik.service: Failed with result 'exit-code'.
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: traefik.service: Service hold-off time over, scheduling restart.
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: traefik.service: Scheduled restart job, restart counter is at 5.
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: Stopped traefik.service.
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: traefik.service: Start request repeated too quickly.
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: traefik.service: Failed with result 'exit-code'.
Feb 18 14:28:38 ip-10-0-2-138 systemd[1]: Failed to start traefik.service.
I should add that I see lighthttpd service is up as I see the landing page at the root url.
Thank you
Never mind.. a colleague had installed lighttpd and failed to mention it to me. I just thought it was part of jupyterhub. I removed it and things are back to normal. Sorry for the false alarm!
-Srini
I am also getting the similar issue but I cannot remove the program that uses the default 80 port. Is there any alternative solution for my issue.
I am also getting the similar issue but I cannot remove the program that uses the default 80 port. Is there any alternative solution for my issue.
Tljh allows you to change the default port 80. The docs ask to be cautious - http://tljh.jupyter.org/en/latest/topic/tljh-config.html#ports
@GeorgianaElena Your advise worked for me to restart the hub using sudo tljh-config reload hub
Thanks @shireenrao
@GeorgianaElena @ParkvilleData same problem, could you help me solve it? Thanks.
@minrk the same problem, I can not visit my tljh with https://xxxx.xxxx.online, could you help me solve it?
my env
/opt/tljh/hub/bin/python -m pip list
Package Version
-------------------------------- ----------
aiohttp 3.6.2
alembic 1.4.2
async-generator 1.10
async-timeout 3.0.1
attrs 19.3.0
backoff 1.10.0
bcrypt 3.1.7
certifi 2020.4.5.1
certipy 0.1.3
cffi 1.14.0
chardet 3.0.4
cryptography 2.9.2
decorator 4.4.2
entrypoints 0.3
escapism 1.0.1
etcd3 0.12.0
grpcio 1.28.1
idna 2.9
idna-ssl 1.1.0
importlib-metadata 1.6.0
ipython-genutils 0.2.0
Jinja2 2.11.2
jsonschema 3.2.0
jupyter-telemetry 0.1.0
jupyterhub 1.1.0
jupyterhub-dummyauthenticator 0.3.1
jupyterhub-firstuseauthenticator 0.14.1
jupyterhub-idle-culler 1.0
jupyterhub-ldapauthenticator 1.3.0
jupyterhub-nativeauthenticator 0.0.5
jupyterhub-systemdspawner 0.13
jupyterhub-tmpauthenticator 0.6
jupyterhub-traefik-proxy 0.1.5
ldap3 2.7
Mako 1.1.2
MarkupSafe 1.1.1
multidict 4.7.5
oauthenticator 0.10.0
oauthlib 3.1.0
onetimepass 1.0.1
pamela 1.0.0
passlib 1.7.2
pip 20.1.1
pkg-resources 0.0.0
pluggy 0.13.1
prometheus-client 0.7.1
protobuf 3.11.3
pyasn1 0.4.8
pycparser 2.20
pycurl 7.43.0.5
pyOpenSSL 19.1.0
pyrsistent 0.16.0
python-consul 1.1.0
python-dateutil 2.8.1
python-editor 1.0.4
python-json-logger 0.1.11
requests 2.23.0
ruamel.yaml 0.15.100
setuptools 39.0.1
six 1.14.0
SQLAlchemy 1.3.16
tenacity 6.2.0
the-littlest-jupyterhub 0.1
toml 0.10.0
tornado 6.0.4
traitlets 4.3.3
typing-extensions 3.7.4.2
urllib3 1.25.9
yarl 1.4.2
zipp 3.1.0
my config
sudo tljh-config show
users:
admin:
- xxxxx
- xxxxxx
allowed: xxxxx
http:
port: 80
auth:
GitHubOAuthenticator:
client_id: xxxxxxxxx
client_secret: xxxxxxxxxxxx
oauth_callback_url: https://xxxx.xxxx.online/hub/oauth_callback
type: oauthenticator.github.GitHubOAuthenticator
FirstUseAuthenticator:
create_users: true
user_environment:
default_app: jupyterlab
https:
enabled: true
letsencrypt:
email: xxxxxxxxx@foxmail.com
domains:
- xxxx.xxxx.online
port: 443
sudo tljh-config reload
active
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
Hub reload with new configuration complete
sudo lsof -i -P -n | grep LISTEN.
systemd-r 341 systemd-resolve 13u IPv4 14687 0t0 TCP 127.0.0.53:53 (LISTEN)
sshd 793 root 3u IPv4 17477 0t0 TCP *:22 (LISTEN)
pure-ftpd 4620 root 6u IPv4 3453099 0t0 TCP *:21 (LISTEN)
pure-ftpd 4620 root 7u IPv6 3453100 0t0 TCP *:21 (LISTEN)
traefik 27875 root 3u IPv6 4364834 0t0 TCP *:80 (LISTEN)
traefik 27875 root 5u IPv6 4365545 0t0 TCP *:443 (LISTEN)
traefik 27875 root 6u IPv4 4365546 0t0 TCP 127.0.0.1:8099 (LISTEN)
python3 28038 root 8u IPv4 4367606 0t0 TCP 127.0.0.1:15001 (LISTEN)
mysqld 31140 mysql 18u IPv6 3417530 0t0 TCP *:3306 (LISTEN)sudo journalctl -u jupyterhub
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z systemd[1]: Started JupyterHub.
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.934 JupyterHub app:2240] Running JupyterHub version 1.1.0
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.935 JupyterHub app:2271] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.1.
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.935 JupyterHub app:2271] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.1.
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.935 JupyterHub app:2271] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.936 JupyterHub app:1394] Writing cookie_secret to /jupyterhub_cookie_secret
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.957 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.957 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
Apr 24 18:08:25 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:25.966 alembic.runtime.migration migration:515] Running stamp_revision -> 4dc2d5a8c53
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:26.035 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [W 2020-04-24 18:08:26.036 JupyterHub app:1624] No admin users, admin interface will be unavailable.
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [W 2020-04-24 18:08:26.036 JupyterHub app:1626] Add any administrative users to `c.Authenticator.admin_use
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:26.036 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowe
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:26.072 JupyterHub app:2311] Initialized 0 spawners in 0.001 seconds
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [W 2020-04-24 18:08:26.075 JupyterHub proxy:643] Running JupyterHub without SSL. I hope there is SSL term
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [I 2020-04-24 18:08:26.075 JupyterHub proxy:646] Starting proxy @ http://:8000
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: [E 2020-04-24 18:08:26.091 JupyterHub app:2718]
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: Traceback (most recent call last):
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: File "/opt/jupyterhub/lib/python3.6/site-packages/jupyterhub/app.py", line 2716, in launch_instance_
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: await self.start()
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: File "/opt/jupyterhub/lib/python3.6/site-packages/jupyterhub/app.py", line 2524, in start
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: await self.proxy.get_all_routes()
Apr 24 18:08:26 iZuf6cf56k6tjepqdcfiv1Z jupyterhub[31393]: File "/opt/jupyterhub/lib/python3.6/site-packages/jupyterhub/proxy.py", line 806, in get_all_routes
...skipping...
May 26 18:51:23 ubuntu18 python3[31747]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 114, in _check_for_traefik_en
May 26 18:51:23 ubuntu18 python3[31747]: resp = await self._traefik_api_request(path)
May 26 18:51:23 ubuntu18 python3[31747]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 157, in _traefik_api_request
May 26 18:51:23 ubuntu18 python3[31747]: auth_password=self.traefik_api_password,
May 26 18:51:23 ubuntu18 python3[31747]: tornado.httpclient.HTTPClientError: HTTP 404: Not Found
May 26 18:51:23 ubuntu18 python3[31747]:
May 26 18:51:23 ubuntu18 python3[31747]: [E 2020-05-26 18:51:23.920 JupyterHub proxy:117] Error checking traefik api for backend /
May 26 18:51:23 ubuntu18 python3[31747]: Traceback (most recent call last):
May 26 18:51:23 ubuntu18 python3[31747]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 114, in _check_for_traefik_en
May 26 18:51:23 ubuntu18 python3[31747]: resp = await self._traefik_api_request(path)
May 26 18:51:23 ubuntu18 python3[31747]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 157, in _traefik_api_request
May 26 18:51:23 ubuntu18 python3[31747]: auth_password=self.traefik_api_password,
May 26 18:51:23 ubuntu18 python3[31747]: tornado.httpclient.HTTPClientError: HTTP 404: Not Found
May 26 18:51:23 ubuntu18 python3[31747]:
May 26 18:51:23 ubuntu18 python3[31747]: [E 2020-05-26 18:51:23.956 JupyterHub proxy:117] Error checking traefik api for backend /
May 26 18:51:23 ubuntu18 python3[31747]: Traceback (most recent call last):
May 26 18:51:23 ubuntu18 python3[31747]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 114, in _check_for_traefik_en
May 26 18:51:23 ubuntu18 python3[31747]: resp = await self._traefik_api_request(path)
May 26 18:51:23 ubuntu18 python3[31747]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 157, in _traefik_api_request
May 26 18:51:23 ubuntu18 python3[31747]: auth_password=self.traefik_api_password,
May 26 18:51:23 ubuntu18 python3[31747]: tornado.httpclient.HTTPClientError: HTTP 404: Not Found
May 26 18:51:23 ubuntu18 python3[31747]:
May 26 18:51:24 ubuntu18 python3[31747]: [I 2020-05-26 18:51:24.001 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 19.51ms
May 26 18:51:26 ubuntu18 python3[31747]: [I 2020-05-26 18:51:26.067 JupyterHub app:2631] JupyterHub is now running at http://:8000
sudo journalctl -u traefik
May 26 18:51:22 ubuntu18 systemd[1]: Started traefik.service.
May 26 18:51:22 ubuntu18 traefik[31744]: time="2020-05-26T18:51:22+08:00" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
May 26 18:51:22 ubuntu18 traefik[31744]: time="2020-05-26T18:51:22+08:00" level=info msg="Traefik version v1.7.18 built on 2019-09-26_01:56:30PM"
May 26 18:51:22 ubuntu18 traefik[31744]: time="2020-05-26T18:51:22+08:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this fe
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Preparing server https &{Address::443 TLS:0xc000587050 Redirect:<nil> Auth:
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Starting server on :443"
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS:<nil> Redirect:<nil>
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Starting server on 127.0.0.1:8099"
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:0xc000951140 Auth:<n
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Starting provider configuration.ProviderAggregator {}"
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Starting server on :80"
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Starting provider *file.Provider {\"Watch\":true,\"Filename\":\"rules.toml\
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Starting provider *acme.Provider {\"Email\":\"stephen_a2z@foxmail.com\",\"A
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Testing certificate renew..."
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Server configuration reloaded on :80"
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Server configuration reloaded on :443"
May 26 18:51:23 ubuntu18 traefik[31744]: time="2020-05-26T18:51:23+08:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
May 26 18:51:24 ubuntu18 traefik[31744]: time="2020-05-26T18:51:24+08:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
May 26 18:51:24 ubuntu18 traefik[31744]: time="2020-05-26T18:51:24+08:00" level=info msg="Server configuration reloaded on :80"
May 26 18:51:24 ubuntu18 traefik[31744]: time="2020-05-26T18:51:24+08:00" level=info msg="Server configuration reloaded on :443"
May 26 18:51:25 ubuntu18 traefik[31744]: time="2020-05-26T18:51:25+08:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
May 26 18:51:25 ubuntu18 traefik[31744]: time="2020-05-26T18:51:25+08:00" level=info msg="Server configuration reloaded on :80"
May 26 18:51:25 ubuntu18 traefik[31744]: time="2020-05-26T18:51:25+08:00" level=info msg="Server configuration reloaded on :443"
finally I find it caused by firewall
so
sudo ufw allow 443
solved my problem
you can use
sudo ufw status
check your firewall.
maybe the words is helpful to you
I'm closing this, it seems to have been caused by a failure to route network traffic to the instance. See #726 regarding a checklist of improvements to make in the documentation to better catch these kinds of issues early on.
what worked for me was:
rm -f /tljh/hub//bin/jupyterhub_cookie_secret
and start again the server:
sudo systemctl restart jupyterhub
I checked the logs:
sudo journalctl -u jupyterhub
and it said that "jupyterhub_cookie_secret" was curropted or had some permissions issue (chmod). I tried to reduce permissions to the bare minimum (rrr/xxx etc), but it only worked when I deleted this file. TLJH will create this file automatically upon reboot.
Hope it will help someone!