Improved debug on migrations
Closed this issue · 2 comments
There are different mandatory and optional input parameters to a scylla migration.
It would be useful at the beginning of migration to dump out input parameters and actual parameters used.
For example, aws region for DynamoDB, scanSegments.
Right now, it is guess work what these parameters are and if they were visible, the diagnosis of problems would be much faster.
I reviewed the way we use the configuration parameters and discovered that some of them are ineffective (e.g., setting scanSegments in the target does nothing, it works only if set to the source table).
Other than that, there are already logs that describe which parameters are used to configure the Hadoop job properties. See e.g. these logs or these logs.
@pdbossman would you consider this issue fixed with the following plan?
- Remove the unused configuration parameters, which currently contribute to your confusion,
- Add more logs about characteristics of the migration that come from the config vs characteristics that are computed with default values (e.g here the supplied AWS region is used)
Or, is there anything else you need to improve debugging on migrations?