Cluster in an unhealthy state

[zerosoul13]

Hello everyone,

Our team has been working on running the go-graphite stack on Kubernetes. We are running the following services:

• carbon-relay-ng
• go-carbon
• carbonapi

We are looking to rebalance our cluster go-graphite cluster. We found buckytools but haven't gotten too far since the tool always reports the cluster in an unhealthy state

/ $ /usr/sbin/bucky servers
Buckd daemons are using port: 4242
Hashing algorithm: [carbon: 5 nodes, 100 replicas, 500 ring members go-carbon-0.go-carbon.goal:2004=None go-carbon-1.go-carbon.goal:2004=None go-carbon-2.go-carbon.goal:2004=None go-carbon-3.go-carbon.goal:2004=None go-carbon-4.go-carbon.goal:2004=None]
Number of replicas: 100
Found these servers:
        go-carbon-0.go-carbon.goal
        go-carbon-1.go-carbon.goal
        go-carbon-2.go-carbon.goal
        go-carbon-3.go-carbon.goal
        go-carbon-4.go-carbon.goal

Is cluster healthy: false
2021/10/28 21:29:14 Cluster is inconsistent.

Could you advice how to bring the cluster to a healthy state?

/ $ bucky -version
bucky <sub-command> [options]
Copyright 2015 - 2017 42 Lines, Inc
Original Author: Jack Neely <jjneely@42lines.net>
Version: 0.4.2

        Bucky is a CLI designed to work with large consistent hashing
        Graphite clusters that have the buckyd daemon installed.  Sub-
        commands will allow you to perform high level operations such
        as backups and restores of specific metrics, backfilling, and
        even rebalancing.

        Use the "help" sub-command for available commands.

[zerosoul13]

To solve the issue, I updated the go-carbon addresses to only use the fqdn and not include the 2004 (Pickle) port