通过docker-compose编排借助docker stack部署服务启动顺序的问题，未等待DB完全启动导致整体部署失败

Question

通过docker-compose编排借助docker stack部署服务启动顺序的问题，未等待DB完全启动导致整体部署失败

luochenxi opened this issue 6 years ago · 7 comments

嗨，你好。@songcser
问题描述：通过docker-compose编排借助docker stack部署服务启动顺序的问题，未等待DB完全启动从而导致整体部署失败。
操作流程：运行examples目录下 ./develop/cluster.sh脚本。
错误日志: 运行docker service logs ms_user_server 得到如下错误

ms_user_server.2.qatzo70vl9bq@ms-node2    | [2018-11-17 08:10:35 +0000] [1] [DEBUG]
ms_user_server.2.qatzo70vl9bq@ms-node2    |                  ▄▄▄▄▄
ms_user_server.2.qatzo70vl9bq@ms-node2    |         ▀▀▀██████▄▄▄       _______________
ms_user_server.2.qatzo70vl9bq@ms-node2    |       ▄▄▄▄▄  █████████▄  /                 \
ms_user_server.2.qatzo70vl9bq@ms-node2    |      ▀▀▀▀█████▌ ▀▐▄ ▀▐█ |   Gotta go fast!  |
ms_user_server.2.qatzo70vl9bq@ms-node2    |    ▀▀█████▄▄ ▀██████▄██ | _________________/
ms_user_server.2.qatzo70vl9bq@ms-node2    |    ▀▄▄▄▄▄  ▀▀█▄▀█════█▀ |/
ms_user_server.2.qatzo70vl9bq@ms-node2    |         ▀▀▀▄  ▀▀███ ▀       ▄▄
ms_user_server.2.qatzo70vl9bq@ms-node2    |      ▄███▀▀██▄████████▄ ▄▀▀▀▀▀▀█▌
ms_user_server.2.qatzo70vl9bq@ms-node2    |    ██▀▄▄▄██▀▄███▀ ▀▀████      ▄██
ms_user_server.2.qatzo70vl9bq@ms-node2    | ▄▀▀▀▄██▄▀▀▌████▒▒▒▒▒▒███     ▌▄▄▀
ms_user_server.2.qatzo70vl9bq@ms-node2    | ▌    ▐▀████▐███▒▒▒▒▒▐██▌
ms_user_server.2.qatzo70vl9bq@ms-node2    | ▀▄▄▄▄▀   ▀▀████▒▒▒▒▄██▀
ms_user_server.2.qatzo70vl9bq@ms-node2    |           ▀▀█████████▀
ms_user_server.2.qatzo70vl9bq@ms-node2    |         ▄▄██▀██████▀█
ms_user_server.2.qatzo70vl9bq@ms-node2    |       ▄██▀     ▀▀▀  █
ms_user_server.2.qatzo70vl9bq@ms-node2    |      ▄█             ▐▌
ms_user_server.2.qatzo70vl9bq@ms-node2    |  ▄▄▄▄█▌              ▀█▄▄▄▄▀▀▄
ms_user_server.2.qatzo70vl9bq@ms-node2    | ▌     ▐                ▀▀▄▄▄▀
ms_user_server.2.qatzo70vl9bq@ms-node2    |  ▀▀▄▄▀
ms_user_server.2.qatzo70vl9bq@ms-node2    |
ms_user_server.2.qatzo70vl9bq@ms-node2    | [2018-11-17 08:10:35 +0000] [1] [INFO] Goin' Fast @ http://0.0.0.0:8030
ms_user_server.2.qatzo70vl9bq@ms-node2    | Executing <Task pending coro=<before_server_start() running at /usr/local/lib/python3.6/site-packages/sanicms/server.py:46> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fcefda7d4c8>()] created at /usr/local/lib/python3.6/asyncio/tasks.py:341> cb=[run_until_complete.<locals>.<lambda>()] created at /usr/local/lib/python3.6/site-packages/sanic/server.py:503> took 1.004 seconds
ms_user_server.2.qatzo70vl9bq@ms-node2    | {"hostname": "f6fd17cc4c04", "name": "sanic", "args": [], "levelname": "INFO", "levelno": 20, "pathname": "/usr/local/lib/python3.6/site-packages/sanicms/service.py", "filename": "service.py", "module": "service", "exc_info": null, "exc_text": null, "stack_info": null, "lineno": 104, "funcName": "service_watcher", "created": 1542442242.8082986, "msecs": 808.2985877990723, "relativeCreated": 6910.207986831665, "thread": 140527157484360, "threadName": "MainThread", "processName": "MainProcess", "process": 20, "message": "service watcher...", "index": "sanic", "document_type": "sanic", "@version": 1, "@timestamp": "2018-11-17T08:10:42.808Z"}
ms_user_server.2.qatzo70vl9bq@ms-node2    | [2018-11-17 08:10:42 +0000] [20] [ERROR] Experienced exception while trying to serve
ms_user_server.2.qatzo70vl9bq@ms-node2    | Traceback (most recent call last):
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 730, in run
ms_user_server.2.qatzo70vl9bq@ms-node2    |     serve(**server_settings)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 609, in serve
ms_user_server.2.qatzo70vl9bq@ms-node2    |     trigger_events(before_start, loop)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 503, in trigger_events
ms_user_server.2.qatzo70vl9bq@ms-node2    |     loop.run_until_complete(result)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "uvloop/loop.pyx", line 1446, in uvloop.loop.Loop.run_until_complete
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanicms/server.py", line 46, in before_server_start
ms_user_server.2.qatzo70vl9bq@ms-node2    |     app.db = await ConnectionPool(loop=loop).init(app.config['DB_CONFIG'])
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanicms/db.py", line 142, in init
ms_user_server.2.qatzo70vl9bq@ms-node2    |     self._pool = await create_pool(**config, loop=self._loop, max_size=100)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 398, in _async__init__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     await self._initialize()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 415, in _initialize
ms_user_server.2.qatzo70vl9bq@ms-node2    |     await first_ch.connect()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 125, in connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     self._con = await self._pool._get_new_connection()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 461, in _get_new_connection
ms_user_server.2.qatzo70vl9bq@ms-node2    |     **self._connect_kwargs)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 1688, in connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     max_cacheable_statement_size=max_cacheable_statement_size)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 551, in _connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     raise last_error
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 543, in _connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     connection_class=connection_class)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 513, in _connect_addr
ms_user_server.2.qatzo70vl9bq@ms-node2    |     connector, timeout=timeout, loop=loop)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return fut.result()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "uvloop/loop.pyx", line 1879, in create_connection
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "uvloop/loop.pyx", line 1858, in uvloop.loop.Loop.create_connection
ms_user_server.2.qatzo70vl9bq@ms-node2    | ConnectionRefusedError: [Errno 111] Connection refused
ms_user_server.2.qatzo70vl9bq@ms-node2    | {"hostname": "f6fd17cc4c04", "name": "sanic.error", "args": [], "levelname": "ERROR", "levelno": 40, "pathname": "/usr/local/lib/python3.6/site-packages/sanic/app.py", "filename": "app.py", "module": "app", "exc_info": ["<class 'ConnectionRefusedError'>", "[Errno 111] Connection refused", "<traceback object at 0x7fcefb7d7688>"], "stack_info": null, "lineno": 735, "funcName": "run", "created": 1542442242.9068415, "msecs": 906.841516494751, "relativeCreated": 7008.750915527344, "thread": 140527157484360, "threadName": "MainThread", "processName": "MainProcess", "process": 20, "message": "Experienced exception while trying to serve", "asctime": "[2018-11-17 08:10:42 +0000]", "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.6/site-packages/sanic/app.py\", line 730, in run\n    serve(**server_settings)\n  File \"/usr/local/lib/python3.6/site-packages/sanic/server.py\", line 609, in serve\n    trigger_events(before_start, loop)\n  File \"/usr/local/lib/python3.6/site-packages/sanic/server.py\", line 503, in trigger_events\n    loop.run_until_complete(result)\n  File \"uvloop/loop.pyx\", line 1446, in uvloop.loop.Loop.run_until_complete\n  File \"/usr/local/lib/python3.6/site-packages/sanicms/server.py\", line 46, in before_server_start\n    app.db = await ConnectionPool(loop=loop).init(app.config['DB_CONFIG'])\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/sanicms/db.py\", line 142, in init\n    self._pool = await create_pool(**config, loop=self._loop, max_size=100)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 398, in _async__init__\n    await self._initialize()\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 415, in _initialize\n    await first_ch.connect()\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 125, in connect\n    self._con = await self._pool._get_new_connection()\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 461, in _get_new_connection\n    **self._connect_kwargs)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connection.py\", line 1688, in connect\n    max_cacheable_statement_size=max_cacheable_statement_size)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py\", line 551, in _connect\n    raise last_error\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py\", line 543, in _connect\n    connection_class=connection_class)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py\", line 513, in _connect_addr\n    connector, timeout=timeout, loop=loop)\n  File \"/usr/local/lib/python3.6/asyncio/tasks.py\", line 358, in wait_for\n    return fut.result()\n  File \"uvloop/loop.pyx\", line 1879, in create_connection\n  File \"uvloop/loop.pyx\", line 1858, in uvloop.loop.Loop.create_connection\nConnectionRefusedError: [Errno 111] Connection refused", "index": "sanic.error", "document_type": "sanic.error", "@version": 1, "@timestamp": "2018-11-17T08:10:42.908Z"}
ms_user_server.2.qatzo70vl9bq@ms-node2    | Traceback (most recent call last):
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/service/server.py", line 28, in <module>
ms_user_server.2.qatzo70vl9bq@ms-node2    |     app.run(host="0.0.0.0", port=app.config['PORT'], debug=True)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 730, in run
ms_user_server.2.qatzo70vl9bq@ms-node2    |     serve(**server_settings)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 609, in serve
ms_user_server.2.qatzo70vl9bq@ms-node2    |     trigger_events(before_start, loop)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 503, in trigger_events
ms_user_server.2.qatzo70vl9bq@ms-node2    |     loop.run_until_complete(result)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "uvloop/loop.pyx", line 1446, in uvloop.loop.Loop.run_until_complete
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanicms/server.py", line 46, in before_server_start
ms_user_server.2.qatzo70vl9bq@ms-node2    |     app.db = await ConnectionPool(loop=loop).init(app.config['DB_CONFIG'])
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/sanicms/db.py", line 142, in init
ms_user_server.2.qatzo70vl9bq@ms-node2    |     self._pool = await create_pool(**config, loop=self._loop, max_size=100)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 398, in _async__init__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     await self._initialize()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 415, in _initialize
ms_user_server.2.qatzo70vl9bq@ms-node2    |     await first_ch.connect()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 125, in connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     self._con = await self._pool._get_new_connection()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 461, in _get_new_connection
ms_user_server.2.qatzo70vl9bq@ms-node2    |     **self._connect_kwargs)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 1688, in connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     max_cacheable_statement_size=max_cacheable_statement_size)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 551, in _connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     raise last_error
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 543, in _connect
ms_user_server.2.qatzo70vl9bq@ms-node2    |     connection_class=connection_class)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return self.gen.send(None)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 513, in _connect_addr
ms_user_server.2.qatzo70vl9bq@ms-node2    |     connector, timeout=timeout, loop=loop)
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "/usr/local/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
ms_user_server.2.qatzo70vl9bq@ms-node2    |     return fut.result()
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "uvloop/loop.pyx", line 1879, in create_connection
ms_user_server.2.qatzo70vl9bq@ms-node2    |   File "uvloop/loop.pyx", line 1858, in uvloop.loop.Loop.create_connection
ms_user_server.2.qatzo70vl9bq@ms-node2    | ConnectionRefusedError: [Errno 111] Connection refused
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | [2018-11-17 08:10:34 +0000] [1] [DEBUG]
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |                  ▄▄▄▄▄
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |         ▀▀▀██████▄▄▄       _______________
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |       ▄▄▄▄▄  █████████▄  /                 \
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |      ▀▀▀▀█████▌ ▀▐▄ ▀▐█ |   Gotta go fast!  |
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |    ▀▀█████▄▄ ▀██████▄██ | _________________/
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |    ▀▄▄▄▄▄  ▀▀█▄▀█════█▀ |/
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |         ▀▀▀▄  ▀▀███ ▀       ▄▄
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |      ▄███▀▀██▄████████▄ ▄▀▀▀▀▀▀█▌
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |    ██▀▄▄▄██▀▄███▀ ▀▀████      ▄██
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | ▄▀▀▀▄██▄▀▀▌████▒▒▒▒▒▒███     ▌▄▄▀
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | ▌    ▐▀████▐███▒▒▒▒▒▐██▌
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | ▀▄▄▄▄▀   ▀▀████▒▒▒▒▄██▀
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |           ▀▀█████████▀
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |         ▄▄██▀██████▀█
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |       ▄██▀     ▀▀▀  █
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |      ▄█             ▐▌
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |  ▄▄▄▄█▌              ▀█▄▄▄▄▀▀▄
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | ▌     ▐                ▀▀▄▄▄▀
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |  ▀▀▄▄▀
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | [2018-11-17 08:10:34 +0000] [1] [INFO] Goin' Fast @ http://0.0.0.0:8030
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | Executing <Task pending coro=<before_server_start() running at /usr/local/lib/python3.6/site-packages/sanicms/server.py:46> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f7169f4a4c8>()] created at /usr/local/lib/python3.6/asyncio/tasks.py:341> cb=[run_until_complete.<locals>.<lambda>()] created at /usr/local/lib/python3.6/site-packages/sanic/server.py:503> took 1.592 seconds
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | {"hostname": "8065c4c7ada4", "name": "sanic", "args": [], "levelname": "INFO", "levelno": 20, "pathname": "/usr/local/lib/python3.6/site-packages/sanicms/service.py", "filename": "service.py", "module": "service", "exc_info": null, "exc_text": null, "stack_info": null, "lineno": 104, "funcName": "service_watcher", "created": 1542442239.5656807, "msecs": 565.680742263794, "relativeCreated": 4876.916408538818, "thread": 140125247527752, "threadName": "MainThread", "processName": "MainProcess", "process": 20, "message": "service watcher...", "index": "sanic", "document_type": "sanic", "@version": 1, "@timestamp": "2018-11-17T08:10:39.565Z"}
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | [2018-11-17 08:10:39 +0000] [20] [ERROR] Experienced exception while trying to serve
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | Traceback (most recent call last):
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 730, in run
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     serve(**server_settings)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 609, in serve
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     trigger_events(before_start, loop)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 503, in trigger_events
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     loop.run_until_complete(result)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "uvloop/loop.pyx", line 1446, in uvloop.loop.Loop.run_until_complete
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanicms/server.py", line 46, in before_server_start
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     app.db = await ConnectionPool(loop=loop).init(app.config['DB_CONFIG'])
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanicms/db.py", line 142, in init
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     self._pool = await create_pool(**config, loop=self._loop, max_size=100)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 398, in _async__init__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     await self._initialize()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 415, in _initialize
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     await first_ch.connect()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 125, in connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     self._con = await self._pool._get_new_connection()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 461, in _get_new_connection
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     **self._connect_kwargs)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 1688, in connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     max_cacheable_statement_size=max_cacheable_statement_size)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 551, in _connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     raise last_error
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 543, in _connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     connection_class=connection_class)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 513, in _connect_addr
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     connector, timeout=timeout, loop=loop)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return fut.result()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "uvloop/loop.pyx", line 1879, in create_connection
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "uvloop/loop.pyx", line 1858, in uvloop.loop.Loop.create_connection
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | ConnectionRefusedError: [Errno 111] Connection refused
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | {"hostname": "8065c4c7ada4", "name": "sanic.error", "args": [], "levelname": "ERROR", "levelno": 40, "pathname": "/usr/local/lib/python3.6/site-packages/sanic/app.py", "filename": "app.py", "module": "app", "exc_info": ["<class 'ConnectionRefusedError'>", "[Errno 111] Connection refused", "<traceback object at 0x7f7167ca35c8>"], "stack_info": null, "lineno": 735, "funcName": "run", "created": 1542442239.687023, "msecs": 687.0229244232178, "relativeCreated": 4998.258590698242, "thread": 140125247527752, "threadName": "MainThread", "processName": "MainProcess", "process": 20, "message": "Experienced exception while trying to serve", "asctime": "[2018-11-17 08:10:39 +0000]", "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.6/site-packages/sanic/app.py\", line 730, in run\n    serve(**server_settings)\n  File \"/usr/local/lib/python3.6/site-packages/sanic/server.py\", line 609, in serve\n    trigger_events(before_start, loop)\n  File \"/usr/local/lib/python3.6/site-packages/sanic/server.py\", line 503, in trigger_events\n    loop.run_until_complete(result)\n  File \"uvloop/loop.pyx\", line 1446, in uvloop.loop.Loop.run_until_complete\n  File \"/usr/local/lib/python3.6/site-packages/sanicms/server.py\", line 46, in before_server_start\n    app.db = await ConnectionPool(loop=loop).init(app.config['DB_CONFIG'])\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/sanicms/db.py\", line 142, in init\n    self._pool = await create_pool(**config, loop=self._loop, max_size=100)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 398, in _async__init__\n    await self._initialize()\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 415, in _initialize\n    await first_ch.connect()\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 125, in connect\n    self._con = await self._pool._get_new_connection()\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/pool.py\", line 461, in _get_new_connection\n    **self._connect_kwargs)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connection.py\", line 1688, in connect\n    max_cacheable_statement_size=max_cacheable_statement_size)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py\", line 551, in _connect\n    raise last_error\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py\", line 543, in _connect\n    connection_class=connection_class)\n  File \"/usr/local/lib/python3.6/asyncio/coroutines.py\", line 110, in __next__\n    return self.gen.send(None)\n  File \"/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py\", line 513, in _connect_addr\n    connector, timeout=timeout, loop=loop)\n  File \"/usr/local/lib/python3.6/asyncio/tasks.py\", line 358, in wait_for\n    return fut.result()\n  File \"uvloop/loop.pyx\", line 1879, in create_connection\n  File \"uvloop/loop.pyx\", line 1858, in uvloop.loop.Loop.create_connection\nConnectionRefusedError: [Errno 111] Connection refused", "index": "sanic.error", "document_type": "sanic.error", "@version": 1, "@timestamp": "2018-11-17T08:10:39.692Z"}
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | Traceback (most recent call last):
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/service/server.py", line 28, in <module>
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     app.run(host="0.0.0.0", port=app.config['PORT'], debug=True)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 730, in run
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     serve(**server_settings)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 609, in serve
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     trigger_events(before_start, loop)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 503, in trigger_events
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     loop.run_until_complete(result)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "uvloop/loop.pyx", line 1446, in uvloop.loop.Loop.run_until_complete
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanicms/server.py", line 46, in before_server_start
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     app.db = await ConnectionPool(loop=loop).init(app.config['DB_CONFIG'])
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/sanicms/db.py", line 142, in init
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     self._pool = await create_pool(**config, loop=self._loop, max_size=100)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 398, in _async__init__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     await self._initialize()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 415, in _initialize
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     await first_ch.connect()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 125, in connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     self._con = await self._pool._get_new_connection()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/pool.py", line 461, in _get_new_connection
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     **self._connect_kwargs)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 1688, in connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     max_cacheable_statement_size=max_cacheable_statement_size)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 551, in _connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     raise last_error
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 543, in _connect
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     connection_class=connection_class)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return self.gen.send(None)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/site-packages/asyncpg/connect_utils.py", line 513, in _connect_addr
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     connector, timeout=timeout, loop=loop)
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "/usr/local/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |     return fut.result()
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "uvloop/loop.pyx", line 1879, in create_connection
ms_user_server.1.2ddjhzlgwnsy@ms-node1    |   File "uvloop/loop.pyx", line 1858, in uvloop.loop.Loop.create_connection
ms_user_server.1.2ddjhzlgwnsy@ms-node1    | ConnectionRefusedError: [Errno 111] Connection refused

我查看develop 存在wait-for-it.sh 和 wait-service.sh脚本，但并未使用。不知为何？
另外，运行 eval $(docker-machine env ms-master) 后再本地请求 http://localhost:9411/zipkin/ ， http://localhost:8090/ 等服务接口并不能成功访问。
盼回，谢谢！

Answer 1 · 2018-11-19T15:11:02.000Z

cluster.sh脚本是建立集群部署的，是后来做的。wait-for-it.sh和wait-service.sh没有用上，cluster.sh脚本还没来得及做上，现在只能手动启动服务。另外cluster.sh脚本跑起来之后，要用ms-master的IP访问，如 http://192.168.99.100:9411/zipkin/ 现在还有一些其他问题没有解决，自动注册注册不上了。

Answer 2 · 2018-11-19T15:20:51.000Z

@songcser 盼早日fixed这些问题，至少一个dome可以顺利跑起来。很棒的一个项目。期望中，加油！

Answer 3 · 2018-11-25T16:05:45.000Z

好想是docker集群的问题，端口调不通了

Answer 4 · 2018-11-26T02:03:04.000Z

@songcser 是docker swarm 自身集群的问题吗？若用k8s部署是否可行呢？

Answer 5 · 2018-11-26T15:31:02.000Z

用k8s是可以的，就不需要服务发现和自动注册了，k8s这些都做了

Answer 6 · 2018-11-30T15:56:24.000Z

可以使用18.06版本的boot2docker，这个是可以访问的。https://github.com/boot2docker/boot2docker/releases/tag/v18.06.1-ce.

Answer 7 · 2018-12-01T07:29:45.000Z

@songcser 好棒!! 已check成功work. 完美！！，持续关注。