mallocator/Elasticsearch-Exporter

eexport.sh is broken

iamjochem opened this issue · 1 comments

iamjochem commented

hi,

I've been checking out your tool (master @ 7bb66fa) just now and noticed that the eexport.sh wrapper script is broken, some of the errors I encountered were:

  1. TypeError: listener must be a function - caused by exporter.printSummary not being a function.
  2. TypeError: Cannot read property 'run' of undefined - caused by exporter.main.run (which I believe should be changed to exporter.run)

If I fix those errors then all I get is the message Starting data export after which the process promptly exits (with 0 as an exit code), without inserting any documents

i.e.

  • ./eexport.sh -sh localhost -si some_index -th 192.168.99.100 - this imports nothing
  • node ./exporter.js -sh localhost -si some_index -th 192.168.99.100 - this imports something

from what I can tell, atm, the wrapper shell-script is lagging behind with respect to your 2.0 development of the tool itself.

This maybe relevant: my indexes & types are already setup on the destination machine/cluster - all the [index & type] names correspond but there are differences, I am trying to migrate data from a 1.3 instance to a 2.0 instance for testing purposes and the mappings in my ES1.3 instance are not valid for ES2.x. Looking at the source code (here) is seems that this is a use case not explicitly supported (indexes are not PUT if they already exist in the destination but the same check is not performed for mappings)

thanks for reading!

side note: when running exporter.js I notice that the progress output in the console is showing nonsense, all I get, right from the start is Processed 139572 of 139072 entries (100%), processing more docs than exist in the index seems unlikely! (139072 is the correct number of docs in the index I am testing with), additionally I am noticing that substantial number of documents are not inserted/migrated (only _65499) of 139072) ... this is something I will try to look into in the next day or two (and hopefully create a PR or two for).

mallocator commented

Thanks for the extensive report. Right now I'm focusing on other projects which is why there hasn't been much development in the last few months, but if you submit a PR I will be sure to review it asap.

I haven't looked at the changes that come with 2.0, but the code is written with multiple APIs in mind. At some point I will implement a proper translation from 1.x to 2.x, but no promise on when that will be.
Sorry.