File export/import not functionnal

[PMickael]

Hi,

There are problems with file import method.
https://github.com/mallocator/Elasticsearch-Exporter/blob/master/drivers/file.driver.js#L277

You cannot use callback twice in the same function, so first work and the next one throw an error, it can't be used like a yield.

#is not accurate because the file export start with a \n
while (buffer.indexOf('\n') > 0) 
#should be 
while (buffer.indexOf('\n') != -1) {
    var endOfLine = buffer.indexOf('\n');
    var line = buffer.substr(0, endOfLine).trim();
    buffer = buffer.substr(endOfLine + 1);
    if (line != '\n' && line != '') {
       log.info(env.options.run.step);
       log.info(items.length);
       try {
           items.push(JSON.parse(line));
       } catch (e) {
            log.error(e);
            log.debug("File driver couldn't read JSON line from source " + line);
            log.error(line);
            process.exit(1);
        }
    }

But that not good, because callback can be call only 1 time.

[mallocator]

Why? Callback is a function and that can be called as many times as you want. What's happening here is that every time a batch of entries has been fetched it is passed on to be processed.

Now I haven't looked at the code in a while so I'm not 100% sure that this is the desired effect, but I have run tests where these operations work, so I'm pretty sure that this works as intended.

Unless there's further evidence that this is actually a bug, I'm closing this for now.

[PMickael]

Why ? I don't actually know but, this function throw "[Error: Callback was already called.]"

The test case work, because of :

while (buffer.indexOf('\n') > 0) 

He never go in this function, because the exported file begin by a \n so = 0.

[mallocator]

Okay, maybe a little more detail could help shed some light on the core problem.

When do you see this error occur? Are you on the latest version of EE? What parameters are you using to cause this error? If you enable verbose logging -v and show me the output, I can probably figure out more about what's wrong. If I can reproduce the exception, I can fix it.

In any case thanks for taking the time to report this.

[PMickael]

Yes,

There the patch I made to find this problem :
https://gist.github.com/PMickael/affc885d0e1e11161ac9

nodejs --version
v5.7.1

[mallocator]

I looked at your patch, but that still doesn't help me determine where the problem stems from. Do you have a stack trace that I can look at? Are you running your own unit tests? Quite honestly I can't reproduce the error here.

[PMickael]

Hi,

I will try to help as much as I can.
Here the command line :

nodejs exporter.js -v -si index_name -s file -sf backup -st type -th xxx.xxx.xxx.xxx -ti index_name -tt type -rm false

There my debug log : https://gist.github.com/PMickael/25d2695af46129a68680

We can see the first time callback is call, the counter back to 0 at line 356 of the log file, and the next time he try to call callback, an error is raise.

I'm sorry I can't give you my data file, but is working well, I haven't any problem to reimport my data in python.
First time I try to reimport the data, I saw the problem because my data file is 81Go, and he try to load every line at one with while (buffer.indexOf('\n') > 0) and it's why I update the code of the exporter and found this bug.

To find more information search [Error: Callback was already called.] on Google, I think it's need to be rewrite with yield instead of a callback function.

Mickael