anlek/mongify

Using Sync Command Facing Timeout Issue & batch_size is not Working.

Opened this issue · 6 comments

kamrankhatti commented

Hi -
When I run process command data migration works perfectly, but when I use sync command batch_size 10000 does not work (facing weird issue batch_size works for embedded tables but not for others) hence it causes TinyTds::Error: Adaptive Server connection timed out error in some tables which has bunch of records.

Also the migration process is very slow when we use sync as compare to process command, I think this same fix #38 should implement for sync as well. Need @hammady consideration over it.

Thanks
Kamran

srizviOfficial commented

@kamrankhatti Thanks for identifying this issue

@hammady @anlek

I am facing this issue, Process command & sync commands behaves differently.

I am able to write translation file which perfectly translate data from MSSQL SERVER to MongoDB via "process command"without much issue's there are some but we able to fix them.

But, we have requirement to deal with differential records (delta) as well. For that we are using Sync command to translate data but it has some issue which we need to fix. There are some timeout error other thing which IO observe is that its not migrating data in chunks instead its processing whole record at once which making issue's for us.

@hammady do you recommend any work around to work with or how I can deal this issue

hammady commented

If the process command was updated to work in batches after I merged the sync feature, I should be able to port this feature as well. @anlek please confirm.

srizviOfficial commented

@hammady That will be great help man

anlek commented

Hey guys,

I'm not sure, I think the sync command is not using the same commands internally to batch the data.
I think this is the part of the code that is not being batched.

@hammady Does that sound right to you?

kamrankhatti commented

One more thing I found in sync command it ignores all embedded tables, each time you run sync command it brings all data for embedded tables.

I have noticed it does not insert embedded table names in __mongify_sync_helper__ table thats why each time sync command brings all data for embedded tables.

@hammady had you tested sync for embedded tables?

hammady commented