Some thoughts on slice_task
Closed this issue · 5 comments
We can use IndexLauncher in the code to start some tasks
Assume that our CPU performance is relatively powerful and the region is relatively large. When I write the code, the partition is only divided into 4 parts.
How to use slice_task or other functions in mapper to divide the current 4 parts into finer parts, such as into 16 parts or 64 parts, etc.
I think there may be some misunderstanding here.
slice_task does not (and cannot) change the number of tasks in an IndexLauncher. If you launch 4 tasks, you will always get 4 tasks. If you launch 16 tasks, you will always get 16 tasks. Nothing in the mapper can change that.
What slice_task does is take an index launch and decide how it gets split up over the machine. It can do so recursively. But the number of tasks in the index launch is fixed and does not change.
If you want the mapper to decide how finely to partition your problem, what you want is the Tunable interface. Application code can query the mapper for a Tunable. The mapper could respond with 4 or 16 or 64, depending on e.g., how big the machine is and how many cores there are. Then the application takes that, does an index launch with that number of tasks, and then the mapper can choose where to place them.
@elliottslaughter Thank you for your answer. You already understand my meaning
After using the indexlauncher to launch the task that splits a partition into 4 parts using the create_equal_partition function, I want to perform the task of splitting the 4 partitions into 16-partitions,32-partitions, and so on inside the mapper. How should I do this, could you briefly describe the steps and provide some examples of similar cases?
Mappers don't create partitions either.
Typically, the application would query a tunable that would return the N that tells you how many subregions to make. The application then passes that to create_equal_partition (or whatever) to make the corresponding partition.
Does that make sense?
@97CBR Do you have any further questions?
I have no more questions. Thank you for your patient replies. Thank you very much.