Tracking iterator progress
kemsky opened this issue · 4 comments
Currently, example contains the following code:
var lastFromModifiedDate = response.CustomerRet.OrderBy(m => m.TimeModified).Select(m => m.TimeModified).LastOrDefault();
await dbContext.SaveIfNewerAsync(LAST_MODIFIED_CUSTOMER, lastFromModifiedDate);
I've noticed that entities received in response are not sorted and in case sync is interrupted next sync can skip some records.
Example. The first batch has lastFromModifiedDate = 01.01.2020
, the second batch lastFromModifiedDate = 01.01.2019
, third batch lastFromModifiedDate = 21.05.2020
. So LAST_MODIFIED_CUSTOMER
is now 21.05.2020, but there are few more batches pending (i.e. from 2018). If system crashes or connection drops - next sync will set request.FromModifiedDate = 21.05.2020
... and miss all remaining items.
Is there any way to track sync in a safe manner?
Just to be clear, I have never seen batches being out of order in the same CustomerQuery
. Are you seeing something else here?
Regarding connection drops or system crashes. I think you might miss a couple yes, but not an entire catalog (if my first assumption is valid). I think you would have to implement flags to be fully resilient for such thing. Flags in the AuthenticateAsync
and CloseConnectionAsync
. Any exceptions caught with the other methods, you could flag it indicating that the last modified customer might not be correct.
@jsgoupil, I've used customers as an example, actually, I need to sync Bills and I see that TimeModified
is out of order (I'll double-check though).
Currently, I'm going to use two columns to store progress, the first column is updated using SaveIfNewerAsync
, it stores temporary value between iterator calls, the second column is updated only when iteratorRemainingCountValue=0
and stores final value. Can I use iteratorRemainingCountValue
for this purpose?
@kemsky I think that iteratorRemainingCountValue
should work.
But you have to come up with a "solution" if you don't reach that iteratorRemainingCountValue
to 0?
Keep us updated on the BillQuery
. And if you find an elegant solution, we could integrate it in this library.
No actions