Parallel Processing
itdependsnetworks opened this issue · 2 comments
Environment
- DiffSync version: latest
Proposed Functionality
The ability to process parts of the sync that do not effect each other.
This likely requires a facility that developers can describe what can be parallel as well as potentially finding specific scenarios where it can automagically happen.
Use Case
Use cases that would benefit from this include:
- Getting Side A and Side B at the same time
- Getting all of table 1 and all of table 2 from Side A at the same time
- Update the model where there is no relationships at the same time
- A parent of multiple children, each of those children can run at the same time
isn't this something you could infer based on the children relationships? I mean, all the object of one type, e.g site, should be able to be processed in parallel. Obviously, you can find shared children, but this conflict can be postponed for a final merge.
Don't know if assuming, by default, that the objects of parent models can run in parallel makes sense. Then, if the library offers a remediation final step using the already supported Redis memory, you could run multiprocessing.
does it make sense to you? or maybe missing some point?
Some can be inferred, some may not be able to be inferred, but yes, that is the point