m-labs/artiq

Dashboard cannot connect with remote moninj and coreanalyzer proxy

Closed this issue · 5 comments

bodokaiser commented

Bug Report

One-Line Summary

Dashboard cannot connect with remote moninj and coreanalyzer proxy

Issue Details

Steps to Reproduce

  1. Have artiq_master, aqctl_moninj_proxy, and aqctl_coreanalyzer_proxy run on a remote computer, e.g., ctrl.
  2. Start the dashboard from a different computer in the network of ctrl via artiq_dashboard -s ctrl

Expected Behavior

Dashboard successfully connects to moninj and coreanalyzer proxy.

Actual (undesired) Behavior

The terminal output of artiq_dashboard -s ctrl as well as the logging table of the dashboard GUI constantly print error messages, e.g.:

ConnectionRefusedError: [Errno 111] Connect call failed ('::1', 1385, 0, 0)
ERROR:dashboard:artiq.dashboard.moninj:failed to connect to moninj. Is aqctl_moninj_proxy running?
Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/moninj.py", line 725, in mi_connector
    await new_mi_connection.connect(self.mi_addr, self.mi_port)
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_moninj.py", line 32, in connect
    self._reader, self._writer = await async_open_connection(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1086, in create_connection
    raise exceptions[0]
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1070, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 974, in _connect_sock
    await self.sock_connect(sock, address)
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/selector_events.py", line 638, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/selector_events.py", line 678, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('::1', 1383, 0, 0)

Your System (omit irrelevant parts)

  • ARTIQ version: ARTIQ v8.0+554d7bd

Background

I think https://github.com/m-labs/artiq/blob/master/artiq/dashboard/moninj.py#L427 should use the server address from the command line instead of the host from the device_db (?) (or localhost). Same with the coreanalyzer

sbourdeauducq commented

What's your device database?
You need to update the entries for these controllers if it's not just localhost in your setup.

sbourdeauducq commented

Additionally, the controllers need to bind the network address and not just localhost. IIRC that's in the manual.

bodokaiser commented

I updated the device_db.py to

# Autogenerated for the main variant
core_addr = "192.168.0.1"

ctrl_addr = "10.163.100.12"

device_db = {
    "core": {
        "type": "local",
        "module": "artiq.coredevice.core",
        "class": "Core",
        "arguments": {
            "host": core_addr,
            "ref_period": 1e-09,
            "analyzer_proxy": "core_analyzer",
            "target": "rv32g",
            "satellite_cpu_targets": {},
        },
    },
    "core_log": {
        "type": "controller",
        "host": "::1",
        "port": 1068,
        "command": "aqctl_corelog -p {port} --bind {bind} " + core_addr,
    },
    "core_moninj": {
        "type": "controller",
        "host": ctrl_addr,
        "port_proxy": 1383,
        "port": 1384,
        "command": "aqctl_moninj_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
        + core_addr,
    },
    "core_analyzer": {
        "type": "controller",
        "host": ctrl_addr,
        "port_proxy": 1385,
        "port": 1386,
        "command": "aqctl_coreanalyzer_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
        + core_addr,
    },
    "core_cache": {
        "type": "local",
        "module": "artiq.coredevice.cache",
        "class": "CoreCache",
    },
    "core_dma": {"type": "local", "module": "artiq.coredevice.dma", "class": "CoreDMA"},
    "i2c_switch0": {
        "type": "local",
        "module": "artiq.coredevice.i2c",
        "class": "I2CSwitch",
        "arguments": {"address": 0xE0},
    },
    "i2c_switch1": {
        "type": "local",
        "module": "artiq.coredevice.i2c",
        "class": "I2CSwitch",
        "arguments": {"address": 0xE2},
    },
}

// ... peripherals

but now get a timeout and name resolution errors:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/waveform.py", line 78, in _reconnect
    await asyncio.wait_for(new_receiver.connect(self.addr, self.port_proxy),
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError
ERROR:dashboard:artiq.dashboard.moninj:failed to connect to moninj. Is aqctl_moninj_proxy running?
Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/moninj.py", line 725, in mi_connector
    await new_mi_connection.connect(self.mi_addr, self.mi_port)
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_moninj.py", line 32, in connect
    self._reader, self._writer = await async_open_connection(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1420, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/aw9aizaf8cx0x19kjxswink5fgvz84kq-python3.11-qasync-0.24.1/lib/python3.11/site-packages/qasync/__init__.py", line 150, in run
    r = callback(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Temporary failure in name resolution
ERROR:dashboard:artiq.dashboard.waveform:error connecting to analyzer proxy
Traceback (most recent call last):
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_analyzer.py", line 157, in connect
    await keepalive.async_open_connection(host, port)
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1420, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/waveform.py", line 78, in _reconnect
    await asyncio.wait_for(new_receiver.connect(self.addr, self.port_proxy),
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError
ERROR:dashboard:artiq.dashboard.waveform:error connecting to analyzer proxy
Traceback (most recent call last):
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_analyzer.py", line 157, in connect
    await keepalive.async_open_connection(host, port)
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1420, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/waveform.py", line 78, in _reconnect
    await asyncio.wait_for(new_receiver.connect(self.addr, self.port_proxy),
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError
ERROR:dashboard:artiq.dashboard.moninj:failed to connect to moninj. Is aqctl_moninj_proxy running?
Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/moninj.py", line 725, in mi_connector
    await new_mi_connection.connect(self.mi_addr, self.mi_port)
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_moninj.py", line 32, in connect
    self._reader, self._writer = await async_open_connection(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1420, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/aw9aizaf8cx0x19kjxswink5fgvz84kq-python3.11-qasync-0.24.1/lib/python3.11/site-packages/qasync/__init__.py", line 150, in run
    r = callback(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Temporary failure in name resolution
ERROR:dashboard:artiq.dashboard.waveform:error connecting to analyzer proxy
Traceback (most recent call last):
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_analyzer.py", line 157, in connect
    await keepalive.async_open_connection(host, port)
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1420, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/waveform.py", line 78, in _reconnect
    await asyncio.wait_for(new_receiver.connect(self.addr, self.port_proxy),
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError
ERROR:dashboard:artiq.dashboard.moninj:failed to connect to moninj. Is aqctl_moninj_proxy running?
Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/moninj.py", line 725, in mi_connector
    await new_mi_connection.connect(self.mi_addr, self.mi_port)
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_moninj.py", line 32, in connect
    self._reader, self._writer = await async_open_connection(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1420, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/aw9aizaf8cx0x19kjxswink5fgvz84kq-python3.11-qasync-0.24.1/lib/python3.11/site-packages/qasync/__init__.py", line 150, in run
    r = callback(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Temporary failure in name resolution
ERROR:dashboard:artiq.dashboard.waveform:error connecting to analyzer proxy
Traceback (most recent call last):
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/coredevice/comm_analyzer.py", line 157, in connect
    await keepalive.async_open_connection(host, port)
  File "/nix/store/vv837g1qs47j1hgsgvvyb4cyf82g5zjk-python3.11-sipyco-1.7/lib/python3.11/site-packages/sipyco/keepalive.py", line 80, in async_open_connection
    reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1046, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 1420, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/nhxybj21xn8ipaairklzl0jg2jg7fyi7-python3.11-artiq-8.0+554d7bd/lib/python3.11/site-packages/artiq/dashboard/waveform.py", line 78, in _reconnect
    await asyncio.wait_for(new_receiver.connect(self.addr, self.port_proxy),
  File "/nix/store/4rf5qybw37b4lh1g0xczlv14sqdbmnpm-python3-3.11.9/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError

I checked that the proxies are running on the server by telnet 10.163.100.12 <port> where <port> is 1383-1386

When I try to open artiq_dashboard on ctrl which has a direct connection to the core device, I get a "Device or Resource busy" error (I also changed ctrl_addr in device_db.py on the master computer back to ::1 to not go over the network but same error). Telnetting to the ports via ::1 works.

bodokaiser commented

Problem solved. For future reference:

Introduce a new variable to overwrite the host key in the core_moninj and core_analyzer devices in the device_db.py.

Potential pitfalls:

Even artiq_dashboard runs local, it uses the device_db.py from the computer running artiq_master, which might be out of sync, if you haven't commited and pushed changes to the repository used by artiq_master.
(In case you are using a git hook, you need to add artiq_client scan-devices.

Furthermore, the computer running artiq_master was not able to do DNS lookups - although it works in the shell - so we will use IPs for now.

sbourdeauducq commented

artiq_master does not use the repository for the device database, only for experiments.