elasticsearch-index-migrate

This software is a command line tool for version control and mapping changes of Elasticsearch mappings.
Elasticsearch versions 6.x and 7.x (up to 7.7.0) are supported.
You can see a sample of this tool here.

Breaking change
Requirements
Usage
Setting
Migration script
Query parameters
Where to store migration scripts
Commands

Breaking change

-i, --indexName Remove the Flags option in version 0.9 or later. Instead of -i, --indexName Flags option, you can pass the value as a command argument.

Requirements

node.js (>=16.15.1)
npm (>=6.14.7)
elasticsearch 6.x and 7.x

This project has been tested with node.js(v16.15.1, v18.4.0), npm(6.14.7) and elasticsearch(6.8.12 and 7.7.1).

Usage

$ npm install -g elasticsearch-index-migrate
$ elasticsearch-index-migrate COMMAND
running command...
$ elasticsearch-index-migrate (-v|--version|version)
elasticsearch-index-migrate/0.8.3 darwin-x64 node-v16.15.1
$ elasticsearch-index-migrate --help [COMMAND]
USAGE
  $ elasticsearch-index-migrate COMMAND
...

Setting

You need to configure the connection settings and migration settings for this software.
The settings can be written in JSON format or read from environment variables.
A JSON file is read by passing a path to the argument of the executable command. If you set it in an environment variable, the system will read it automatically when you execute the command.
If you put a configuration file named config.json in ~/.config/elasticsearch-index-migrate/, it will be loaded automatically.
The environment variables take precedence over the settings. Then the -O option takes precedence.

If you want to use a JSON file, please use the following.

{
    "elasticsearch": {
        "version": "7",
        "connect": {
            "host": "http://0.0.0.0:9202",
            "sslCa": "",
            "cloudId": "",
            "username": "",
            "password": ""
        }
    },
    "migration": {
        "locations": ["/migration"],
        "baselineVersion": "v1.0.0"
    }
}

elasticsearch.version : Enter the version of Elasticsearch you are using. If you are using OpenSearch, please write 'opensearch'.
elasticsearch.connect.host : Enter the Elasticsearch Host you are using.
elasticsearch.connect.sslCa : Fill in the location of your Elasticsearch SSL certificate.
elasticsearch.connect.cloudId : Enter your Elasticsearch Cloud ID.
elasticsearch.connect.username : Enter your Elasticsearch username.
elasticsearch.connect.password : Enter your Elasticsearch password.
migration.locations : Enter the full path or relative path to the directory where you want to store your migration script
migration.baselineVersion : Please fill in the base version of the migration script

To set the environment variables, set the following variables.

ELASTICSEARCH_MIGRATION_LOCATIONS
ELASTICSEARCH_MIGRATION_BASELINE_VERSION
ELASTICSEARCH_VERSION
ELASTICSEARCH_HOST
ELASTICSEARCH_SSL
ELASTICSEARCH_CLOUDID
ELASTICSEARCH_USERNAME
ELASTICSEARCH_PASSWORD

Migration script

Migration scripts are written in JSON format. File names should follow the format v{version}__{name}.json.

Currently the following migration types are supported:

ADD_FIELD
CREATE_INDEX
DELETE_INDEX
ALTER_SETTING

Migration script format

In the case of ADD_FIELD scripts, describe the mapping information that can be executed with the Put mapping API under the migrate_script property.

{
  "type": "ADD_FIELD",
  "description": "description",
  "migrate_script": {
    "properties": {
      "user_id": {
        "type": "keyword"
      },
      ...
    }
  }
}

In the case of CREATE_INDEX scripts, write a Request body that can be executed with the Create index API under the migrate_script property.

{
  "type": "CREATE_INDEX",
  "description": "description",
  "migrate_script": {
    "settings": {
        "index": {
            "refresh_interval": "1s",
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    },
    "mappings": {
        "properties": {
          "user_id": {
            "type": "keyword"
          },
          ...
        }
    }
  }
}

For a DELETE_INDEX script, do the following.

{
  "type": "DELETE_INDEX",
  "description": "description"
}

In the case of ALTER_SETTING scripts, write a Request body that can be executed with the Update Indices Settings API under the migrate_script property.

{
  "type": "ALTER_SETTING",
  "description": "description",
  "migrate_script": {
    "index" : {
        "number_of_replicas" : 1
    }
  }
}

Query parameters

Query parameters can be specified in JSON format in the migration script. You can use the query parameters according to the migration type. For more information, please check out Elasticsearch.

For Example:

{
  "type": "CREATE_INDEX",
  "description": "description",
  "migrate_script": {
    "settings": {
        "index": {
            "refresh_interval": "1s",
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    },
    "mappings": {
        "properties": {
          "user_id": {
            "type": "keyword"
          },
          ...
        }
    }
  },
  "query_parameters": {
    "include_type_name": true,
    "timeout": 10
  }
}

Where to store migration scripts

Save your migration script to the directory you set up in ELASTICSEARCH_MIGRATION_LOCATIONS or JSON's migration.locations. If the storage location is /elasticsearch/migration, store the script in a directory like the following. This example is for the index names {index_name} and {index_name}_v1, {index_name}_v2. If the index name has a version, it is necessary to separate the directories.For names in the form {index_name}-{version} or {index_name}_{version}, separate the directory from {index_name}/{version}/. In this case, please put the following script in {version}/.

elasticsearch/
  ┗ migration/
    ┣ indices/
    │   ┣ {index_name}/
    │   │  ┣ v1/ 
    │   │  │  ┣ V1.0.0__init_mapping.json
    │   │  │  ┗ V1.0.1__add_field.json 
    │   │  ┣ v2/
    │   │  │  ┣ V2.0.0__init_mapping.json
    │   │  │  ┗ V2.0.1__add_field.json 
    │   │  ┗ v3/
    │   ┗ {index_name}/
    │      ┗ V1.0.0__create_index.json

Commands

elasticsearch-index-migrate baseline [NAME]
elasticsearch-index-migrate clean [NAME]
elasticsearch-index-migrate help [COMMAND]
elasticsearch-index-migrate init
elasticsearch-index-migrate migrate [NAME]
elasticsearch-index-migrate plan [NAME]
elasticsearch-index-migrate recovery [NAME]