gliderlabs/registrator

Support mode -deregister never

itrofimow opened this issue · 1 comments

It's more like an enhancement request, not an issue.

Request might seem strange, but let me explain further:
I'd like to use Consul as my God-Object, the one who knows everything about my setup/infrastructure/services etc.
Furthermore, i want to use Consul as the source of my monitoring/alerting, however this seems currently impossible with registrator unregistering services when containers are stoped gracefully - there are just no information about them left anywhere in Consul, they are just gone and that's not suitable for me.

The most straightforward case is like this:
If at some point all containers of some service are gracefully stoped (my deployment process involves docker stop at some point, for instance, and things might go wrong) then the only way i can know about this is customers complaining about 502-s.

Yes, i could alert about this exact scenario, when all containers of service are dead - just because this gives information (no service with name example anymore at all) but if just some containers are stoped - i cant know about it without taking my infrastructure configuration from somewhere and complicating things matching it against Consul etc.

The workaround for my case is like this: itrofimow@a772b6e
so when container dies its not unregistered and agent tcp-check fails, thus service becomes critical

I am somewhat concerned - may be consul + consul-registrator are just not the suitable duo for this kind of half-static configuration and i am reinventing the bicycle?

Could you please share your thoughts about this and maybe point me at some non-obvious scenarious i am missing (and yes, i do realize that some services might get stale at consul if hostname/container-name/container-ip changes and features relying on deregistering services wont work anymore)

Seems like i've forgot an important explanation about my deployment process:

  • each service is deployed to some static configuration of nodes in each of DC-s
  • hostname:container-name:service-port are always same and dont change for a pair container/node
  • this involves rebalancing load away from one dc, deployment on it, rebalancing load again, deployment on another dc, rebalancing load on both DC-s
  • deployment is basically:
  1. docker stop
  2. docker rm
  3. docker run