salt-formulas/salt-formula-docker

Token unknown with good grains in swarm master

Opened this issue · 6 comments

JohanMoreau commented

Hello,
I try to create a docker swarm. My master seems to be good :

local:
    .......
    docker_swarm_tokens:
        ----------
        manager:
            SWMTKN-1-1wg..............

But on my worker, I can't transfert the token :

docker swarm join --token unknown

I try to add mine like :

mine_functions:
  swarm:
    - mine_function: grains.items

or

mine_functions:
  swarm.docker_swarm_tokens.manager:
    - mine_function: cmd.run
    - 'docker swarm join-token manager -q'

But it doesn't work. Any idea ?

salt 2017.7.4 (Nitrogen)
salt-call 2017.7.4 (Nitrogen)

fpytloun commented

It's using salt mine to share tokens and should be working out of the box when using salt-formula-salt to setup salt master and minions.
Your function that sends grains.items should be ok but you probably need to run this after you restart minion:
salt '*' mine.flush
salt '*' mine.update

JohanMoreau commented

Thanks for your help,
Yes, I had already made these commands, but that does not change anything about my worker.
Is there a way to verify that the minion of the worker gets to have this data (tokens and co) ?
My only way to test now is to make a highstate on the minion and I have trouble understanding the sls. I guess it's the swarm.sls with the join method, but this line is not clear for me :

{%- for node_name, node_grains in salt['mine.get']('*', swarm.mine_function).iteritems() %}
  • 👍1
rendler-denis commented

hello. @JohanMoreau did you manage to track down what it was? I encounter the same issue when deploying to a real environment. it worked ok using salt-ssh deployment on a vagrant env, but everything is off on the real machine. I manually tested the docker installs on both machines and I can connect them but not with salt.

JohanMoreau commented
rendler-denis commented

Hello, @JohanMoreau . I managed to make it work after a few days of pain. In my case, I needed to add the grains.items in the salt mine functions on the minion and after quite a few mine.flush, refresh_grains it seems to be working as expected.
I am using:

Salt Version:
Salt: 2018.3.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: 2.5.1
gitdb: 2.0.3
gitpython: 2.1.8
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5

kmkramer commented

We also hit this issue. There is a chicken egg issue here. When you highstate the custom grains are run first but the commands return empty because docker isn't installed yet. Then when the mine.update runs the grains aren't there.
ie custom grains (saltutil.sync_*) doesn't run again until forced or another highstate is run. I fixed this by adding a sync_grains after swarm_init.

snip..

docker_swarm_init:
  cmd.run:
    - name: >
        docker swarm init
        {%- if swarm.advertise_addr is defined %} --advertise-addr {{ swarm.advertise_addr }}{%- endif %}
        {%- if swarm.get('bind', {}).get('address', None) %} --listen-addr {{ swarm.bind.address }}{% if swarm.bind.port is defined %}:{{ swarm.bind.port }}{% endif %}{%- endif %}
    - unless:
      - "test -e /var/lib/docker/swarm/state.json"
      - "docker node ls | grep -q '{{ grains.nodename }}'"
    - require:
      - service: docker_service
  module.run:
    - name: saltutil.sync_grains
    - refresh: false
    - onlyif:
      - salt-call grains.items | grep docker_swarm

docker_swarm_grains_publish:
  cmd.run:
    - name: "salt-call mine.send {{ swarm.mine_function }}"
    - watch:
      - cmd: docker_swarm_init

We are also working on adding secrets and configs into this formula and hope to submit a PR soon.