Netflix/Priam

Is it possible to use priam only for backup?

Closed this issue · 5 comments

darkpssngr commented

Since Priam does not support vnodes today. I'm planning to maintain the cluster without priam. But is it possible to use Priam only for taking backups?

Thanks in advance.

arunagrawal84 commented

@darkpssngr you can set Priam to not start C* using configuration - priam.cass.manual.start.enable to true. Default value is false. Also, you may want to change the configuration location of cassandra.yaml (via configuraiton - `priam.yamlLocation') so Priam does not modify your .yaml file.

Ensure your backup configurations are pointing correctly and then it should be able to take backups. Note that I have not personally tested this so you might have to give it a try and see if there is anything which is still broken.

Apart from this, there is already a PR from @zmarios in #626 and #623 which actually adds support for vnodes. We will review it and change our infra accordingly before merging to main branch.

darkpssngr commented

Sorry, I didn't get it. Are you saying I should set up the whole cluster with priam and just set priam.cass.manual.start.enable to true and supply my own cassandra.yaml so that it can still use vnodes?

arunagrawal84 commented

Yes, you need to set up the entire cluster with Priam as it can only do the backup on the nodes/instances where it is installed.
Yes, you are correct.

  1. manual.start will ensure that Priam is not interrupting with your C* process management (which I believe is handled somewhere else in your automation suite).
  2. You need to ensure that cassandra.yaml used by C* is not "modified" by Priam as it will change yaml to give only one token range (instead of vnodes).
zmarois commented

The reason I wanted to add vnode support was purely for backup/restore also @darkpssngr . Some other things I learned along the way:

  • If you aren't going to use Priam to start/stop cassandra, you'll need to be able to know if Priam is ready for cassandra to start. In a non-restore scenario, they are pretty much independent, but in a restore scenario, you need to wait for Priam to complete downloading the data files
  • You won't want to use the startup agent (step 6), as it will wait for Priam to be up for cassandra to start, and will set your num tokens to 1, an initial token, and seeds
  • The DeadTokenRetriever might have problems if you try to replace a node, as the replacement node will ask the other live nodes about the token it is replacing, which is not true because your vnodes aren't using the token Priam generates.
  • Your backups will have identifiers that look like tokens (they'd be the token's Priam would have specified)
  • You'll still need to be running in an ASG for Priam to start up. I worked around this in my fork here although I still need to set system property ASG_NAME to my cluster name for Priam to know the cluster name before loading the config.
  • I'm still not 100% confident about the need (or lack of), per step 5 in this overview, to set initial_token when using vnodes. I've documented my thoughts on that in the first paragraph of my comment here. If anyone has more insight into that and it is actually nessessary, Priam would need to at least inform your config of the tokens that were on the node that took the backup, and you'd need Priam for at least that cassandra configuration.
darkpssngr commented

@arunagrawal84 and @zmarois Thanks for the inputs. Since our cluster is going to be a small one 5-6 nodes. We've decided to just use ebs snapshots for now. Do you see any problems with that?