New design for sharding-scaling-core
Opened this issue · 4 comments
After we split whole project to multiple modules #23 , We should also do some redesign for sharding-scaling-core to make it more clear and readable.
My design is
Hi, @KomachiSion ,I read the design carefully, and it is really a good design. But I also have some suggestions:
- I think data migration must pay attention to two points, one is the speed, and the other is the data consistency.
- Distributed system architecture may speed up data migration, but it makes it more difficult to ensure the data consistency.Sometimes,we pay more attention to the data consistency rather than speed.
- In isomorphic system, like migrating data from MySQL to MySQL, I believe no tools can faster than the MySQL official tools like mysqldump or load data.So we use mysqldump in my company to migrate data from MySQL to MySQL. But we also need the migration tools like shardingscaling to migrate realtime data and work with the system to switch data sources.
- Finally, we may refer to other products like datax of alibaba, for data migration, we may have something in common.
In conclusion, we may pay more attention to the data consistency when design.
Distributed system architecture may speed up data migration, but it makes it more difficult to ensure the data consistency.Sometimes,we pay more attention to the data consistency rather than speed.
The master and worker can run in same process, which means they can run without distributed system.
In isomorphic system, like migrating data from MySQL to MySQL, I believe no tools can faster than the MySQL official tools like mysqldump or load data.So we use mysqldump in my company to migrate data from MySQL to MySQL. But we also need the migration tools like shardingscaling to migrate realtime data and work with the system to switch data sources.
It's a custom feature for yours, but it's a nice point for scaling component to make the migrate step to be configurable, users can choose which step they need. But it will be consider in future.
Finally, we may refer to other products like datax of alibaba, for data migration, we may have something in common.
The component is designed for auto-scaling ShardingSphere finally, static migrate data is first and simple version, there will be many different feature.
At last, If you have better design or advise, we are welcome for your contributions.
@KomachiSion ,thank you for your reply, and I believe it will be a greate design 😄