Retries in clustering, when there is an HBase table outage

[timrobertson100]

The GBIF clustering generates an HBase table that is used to add an isInCluster flag on Elasticsearch. To allow a workflow to automate the HBase table replacement, we should implement a retry mechanism that allows a 1-minute outage of the HBase table in the pipelines.

We discussed and chose not to use a more complex solution of creating new tables and watching a ZK node (i.e. like the maps solution) just to keep the implementation simple, knowing this is a weekly or monthly process. We might revisit that decision later.

[muttcg]

Deployed to PROD