Zookeeper backend. randomly not sync files after changes
Opened this issue · 0 comments
itshikanow commented
Hello
I'm setup patroni cluster, and use confd to configure haproxy.
Backend zookeeper
Confd configs structure
/usr/local/etc/confd$ tree
.
├── conf.d
│ └── haproxy.toml
├── confd.toml
└── templates
└── haproxy.tmpl
cat confd.toml
# log-level = "info"
log-level = "debug"
backend = "zookeeper"
confdir = "/usr/local/etc/confd"
watch = true
interval = 1
nodes = ["192.168.50.1:2181",]
noop = false
prefix = "/patroni/15-main"
cat conf.d/haproxy.toml
[template]
# prefix = "/patroni/15-main/"
owner = "haproxy"
mode = "0644"
src = "haproxy.tmpl"
dest = "/usr/local/etc/haproxy/haproxy.cfg"
check_cmd = "/usr/local/sbin/haproxy -c -f {{ .src }}"
reload_cmd = "/bin/bash -c 'true'"
keys = ["/members",]
cat templates/haproxy.tmpl
global
maxconn 1000
log stdout format raw local0
# user haproxy
# group haproxy
defaults
log global
mode tcp
retries 2
timeout client 120m
timeout server 120m
timeout connect 4s
timeout check 5s
# balance leastconn
option tcplog
option log-health-checks
listen stats
mode http
bind *:7000
stats enable
stats uri /
listen master
bind *:5428
option httpchk OPTIONS /master
http-check expect status 200
default-server inter 2s fall 3 rise 2 on-marked-down shutdown-sessions
{{- range gets "/members/*" }}
{{- $data := json .Value }}
server {{base .Key}} {{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 300 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
{{- end }}
listen replicas
bind *:5429
option httpchk OPTIONS /replica?lag=10MB
http-check expect status 200
default-server inter 2s fall 3 rise 2 on-marked-down shutdown-sessions
{{- range gets "/members/*" }}
{{- $data := json .Value }}
server {{base .Key}} {{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 300 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
{{- end }}
{{- range gets "/members/*" }}
{{- $data := json .Value }}
{{- if eq "master" $data.role }}
server {{base .Key}}-master {{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 300 backup
{{- end }}
{{- end }}
start command:
confd -config-file /usr/local/etc/confd/confd.toml
I'm start confd, waiting few seconds. Cond generate haproxy config right. I'm stoping one node of patroni, waiting 1 minute, to disapear znode from zookeeper and config has changed exptectialy, the one node disapear from haproxy config
After this i'm start the patroni node, at first config changed right. But after few seconds config reverted to state without second patroni node while in zookeeper all 2 znode exist and have right data.
If restart confd at this point, the config will regenerate right.
If change interval to from 1 to 5 this behaviour not so frequent, but repeats. I'm need confidence that confd work right.
If use -oneshot and start confd in bash script all work well. But spawn avery 1 second new process is bad.
here the last state of zookeeper after enable patroni node back:
ls /patroni/15-main/members
[devlocal20, devlocal20-2]
get /patroni/15-main/members/devlocal20
{"conn_url":"postgres://192.168.50.200:5440/postgres","api_url":"http://192.168.50.200:8008/patroni","state":"running","role":"replica","version":"3.0.1","xlog_location":4429186584,"timeline":11}
get /patroni/15-main/members/devlocal20-2
{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}
Here the full log of confd with this behaviour
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Backend set to zookeeper
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Starting confd
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Backend source(s) set to 192.168.50.1:2181
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Loading template resources from confdir /usr/local/etc/confd
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Found template: /usr/local/etc/confd/conf.d/haproxy.toml
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Loading template resource from /usr/local/etc/confd/conf.d/haproxy.toml
2023/07/07 13:55:58 Connected to 192.168.50.1:2181
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023/07/07 13:55:58 Authenticated: id=72058294651454940, timeout=4000
2023/07/07 13:55:58 Re-submitting `0` credentials after reconnect
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20:{"conn_url":"postgres://192.168.50.200:5440/postgres","api_url":"http://192.168.50.200:8008/patroni","state":"running","role":"replica","version":"3.0.1","xlog_location":4429186584,"timeline":11} /patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum d41d8cd98f00b204e9800998ecf8427e should be 910dff18d217291b885650402dd3efeb
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg670469573
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum 910dff18d217291b885650402dd3efeb should be 1ef749761af34e16ebe67cff2ae4d849
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg861808736
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20:{"conn_url":"postgres://192.168.50.200:5440/postgres","api_url":"http://192.168.50.200:8008/patroni","state":"stopped","role":"replica","version":"3.0.1"} /patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum 1ef749761af34e16ebe67cff2ae4d849 should be 910dff18d217291b885650402dd3efeb
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg303199807
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum 910dff18d217291b885650402dd3efeb should be 1ef749761af34e16ebe67cff2ae4d849
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg080067730
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2