Dynamic Uploader connection
Felixoid opened this issue · 2 comments
Hello. During huge data migration from the Whisper to ClickHouse I've recognized that the connection to CH is permanent. Because of that, we have an unbalanced loading: only one of 8 CH nodes under an LB receives all data and then redistribute it with a sharding key.
Implementing a mechanism to recognize shouldn't be a hard task, but I'm not sure about the overhead for a new connection per insert. Maybe, implement a config parameter for it?
I assume that the carbon-clickhouse and the graphite-clickhouse are on the same server as the clickhouse. In this configuration, you can use external load balancers for write (carbon-c-relay) and read (carbonapi)
Implementing a mechanism to recognize shouldn't be a hard task, but I'm not sure about the overhead for a new connection per insert. Maybe, implement a config parameter for it?
Overhead will be very small and optional parameter is not required. PR are welcome