Speed of splitting lots of tablets could be improved
keith-turner opened this issue · 1 comments
When the API is called to add splits to a table it currently does the following.
- Create a thread pool of size 16. Code link
- Map provided split points to tablets. Code link
- For each tablet run a split fate operation to completion by creating and waiting on a fate operation. This is done in the thread pool created earlier. Code link
If 1,000 splits are added and those map to 1,000 tablets, then the above algorithm would only work on 16 tablets at a time. If all 1,000 tablets are hosted on tablet servers then it will probably take a while because only 16 will be unassigned and split at a time.
Fate operations always work through the same 4 rpcs, which are begin, execute, wait, and then finish. The above situation could be improved by starting all 1,000 fate operations before waiting on them. Somehow this refactoring should avoid duplicating this exception handling code
If all of the fate split operations could be started before waiting on them, then in the situation of splitting 1000 tablets that were all hosted it would allow all 1,000 to be unassigned and split concurrently in fate.
Closing as completed via #4697