/elasticsearch-migration

A tool to migration elasticsearch index from one cluster to another cluster

Primary LanguagePython

elasticsearch-migration

A tool to migration elasticsearch index from one index to another index (can be different cluster). You can custom the docs that you want to migrate.

Usage

  1. modify conf.json

    {
  "source": {
    "host": "source_host", # source index's es host
    "index": "source_index", # source index
    "type": "source_type", # source type
    "size_per_handle": 500, # doc count per fetch and bulk index
    "scroll_alive": "1m", # scroll alive time
    "dsl": {} # search dsl
  },
  "target": {
    "host": "target_host", # target index's es host
    "index": "target_index", # target index
    "type": "target_type", # target type
    "index_thread_num": 10 # number of thread to bulk index
  },
  "copy_mapping": true # whether need copy mapping from source index to target index, you can set this value false to set target index mapping manually
}

  2. custom docs (if necessary)

    • MigTargetEs#handle_id: custom id. Return None to ignore this doc.
    • MigTargetEs#handle_source: custom source. Return None to ignore this doc.

  3. python run.py

  1%|          | 8000/1381557 [00:15<41:38, 549.85it/s]

Compatibility

It works well under these environment:

  • os: macOS Mojave 10.14
  • python: 3.6.5
  • elasticsearch: 2.2.0, 6.2.3