AnatolyUss/nmig

Infinite loop "loading the data"

snblitz opened this issue · 10 comments

snblitz commented

Cool tool. Just gave it a try. It seems to work fine, but looks to be in an infinite loop.
The data looks to have successfully transferred in a few minutes but for some hours now it continues to print to the screen:
--[loadData] Loading the data...
--[loadData] Loading the data...

snblitz commented

root@mysqlrdsbridge:~/src/nmig# nodejs --version
v10.15.3

AnatolyUss commented
snblitz commented
AnatolyUss commented
snblitz commented
Boshen commented

I'm also getting this.
Out of 304761 rows, it stopped at 303318 rows. In DataLoader, signal.chunks.length is giving me 0 so it infinite looped. Can you guide me on where the data/message is sent from so I can debug further?

Boshen commented

Here's more info:
Everything works fine except it fails on this one table, and it fails on the last chunk.
Somehow I'm getting 41.2 size_in_mb on the last chunk, where every other chunk has a normal 10 size_in_mb.
This makes fillBandwidth in DataPipeManager.ts return an empty array ... which makes it infinite loop.

Here are my last two chunks (omitted some extra fields):

[
  {
    "_offset": 295920,
    "_rowsInChunk": 7398,
    "_rowsCnt": 304761,
    "_id": "41",
    "_size_in_mb": 10,
    "_processed": true
  },
  {
    "_offset": 303318,
    "_rowsInChunk": 7398,
    "_rowsCnt": 304761,
    "_id": "42",
    "_size_in_mb": 41.2,
    "_processed": false
  }
]

I have temporary fixed this by adding the last chunk to dataChunkIndexes when all your checks failed.

    if (dataChunkIndexes.length === 0) {
      dataChunkIndexes.push(conversion._dataPool.findIndex((v) => v._processed === false));
    }
    return dataChunkIndexes;

p.s. great tool though, I couldn't get pgloader to work but your tool worked perfectly.

snblitz commented

I had to time to try the transfer again. All proceeded as last time eventually ending in an infinite loop:

    --[pipeData]  For now inserted: 6639 rows, 
            Total rows to insert into "kayako"."swescalationpaths": 6639
    --[loadData] Loading the data...
    --[pipeData]  For now inserted: 150871 rows, 
            Total rows to insert into "kayako"."swbayeswords": 150871
    --[pipeData]  For now inserted: 22385 rows, 
            Total rows to insert into "kayako"."swticketlinkedtables": 22385
    --[pipeData]  For now inserted: 34735 rows, 
            Total rows to insert into "kayako"."swticketlocks": 34735
    --[loadData] Loading the data...
    --[loadData] Loading the data...
    --[loadData] Loading the data...
    --[loadData] Loading the data...
    --[loadData] Loading the data...
    --[loadData] Loading the data...
    --[loadData] Loading the data...

So far as I can tell all the data transferred. There is no file nmig/logs_directory/errors-only.log

AnatolyUss commented

Hi,
The issue seems to be fixed in a new version (4.0.0), that I've released yesterday.
Sorry, for the delay.
@snblitz and @Boshen , many thanks for your collaboration!

  • 👍1
snblitz commented

I ran version 4.0.0 today and it completed without the infinite loop. :)