A library to migrate Elasticsearch mappings. Inspired by flyway.
Elasticsearch-Evolution executes versioned migration scripts reliable and persists the execution state in an internal Elasticsearch index. Successful executed migration scripts will not be executed again!
- runs on Java 8, 9, 10 and 11
- runs on Spring-Boot 1.5, 2.0 and 2.1 (and of course without Spring-Boot)
- runs on Elasticsearch 6.7.x 6.6.x, 6.5.x, 6.4.x, 6.3.x, 6.2.x
- highly configurable (e.g. location(s) of your migration files, migration files format pattern)
- placeholder substitution in migration scripts
- easily extendable to your needs
- supports microservices / multiple parallel running instances via logical database locks
- ready to use default configuration
- line comments in migration files
First add the latest version of Elasticsearch-Evolution spring boot starter as a dependency in your maven pom.xml:
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>spring-boot-starter-elasticsearch-evolution</artifactId>
<version>0.1.2</version>
</dependency>Elasticsearch-Evolution uses internally Elastics RestHighLevelClient and requires at minimum version 6.6.0. Spring boot uses a older version, so update it in your pom.xml:
<properties>
<elasticsearch.version>6.6.0</elasticsearch.version>
</properties>Place your migration scripts in your application classpath at es/evolution
That's it. Elasticsearch-Evolution runs at application startup and expects your Elasticsearch at http://localhost:9200
First add the latest version of Elasticsearch-Evolution core as a dependency:
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-core</artifactId>
<version>0.1.2</version>
</dependency>Place your migration scripts in your application classpath at es/evolution
Create a ElasticsearchEvolution instance and execute the migration.
// first create a Elastic RestHighLevelClient
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(
RestClient.builder(HttpHost.create("http://localhost:9200")));
// then create a ElasticsearchEvolution configuration and create a instance of ElasticsearchEvolution with that configuration
ElasticsearchEvolution elasticsearchEvolution = ElasticsearchEvolution.configure()
.load(restHighLevelClient);
// execute the migration
elasticsearchEvolution.migrate();A Elasticsearch-Evolutions migration script represents just a rest call. Here is an Example:
PUT _template/my_template
Content-Type: application/json
{
"index_patterns": [
"my_index_*"
],
"order": 1,
"version": 1,
"settings": {
"number_of_shards": 1
},
"mappings": {
"_doc": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 20,
"similarity": "boolean"
},
"locked": {
"type": "boolean"
}
}
}
}
}The first line defines the HTTP method PUT and the relative path to the Elasticsearch endpoint _template/my_template to create a new mapping template.
Followed by a HTTP Header Content-Type: application/json.
After a blank line the HTTP body is defined.
The pattern is strongly oriented in ordinary HTTP requests and consist of 4 parts:
- The HTTP method (required). Supported HTTP methods are
GET,HEAD,POST,PUT,DELETE,OPTIONSandPATCH. The First non-comment line must always start with a HTTP method. - The path to the Elasticsearch endpoint to call (required). The path is separated by a blank from the HTTP method.
You can provide any query parameters like in a ordinary browser like this
/my_index_1/_doc/1?refresh=true&op_type=create - HTTP Header(s) (optional). All non-comment lines after the HTTP method line will be interpreted as HTTP headers. Header name and content are separated by
:. - HTTP Body (optional). The HTTP Body is separated by a blank line and can contain any content you want to sent to Elasticsearch.
Elasticsearch-Evolution supports line-comments in its migration scripts. Every line starting with # or // will be interpreted as a comment-line.
Comment-lines are not send to Elasticsearch, they will be filtered by Elasticsearch-Evolution.
Elasticsearch-Evolution supports named placeholder substitution. Placeholders are marked in your migration script like this: ${my-placeholder}
- starts with
placeholderPrefixwhich is by default${and is configurable. - followed by the
placeholder namewhich can be any string, but must not containplaceholderPrefixorplaceholderSuffix - ends with
placeholderSuffixwhich is by default}and is configurable.
Here is an example filename: V1.0__my-description.http
The filename has to follow a pattern:
- starts with
esMigrationPrefixwhich is by defaultVand is configurable. - followed by a version, which have to be numeric and can be structured by separating the version parts with
. - followed by the
versionDescriptionSeparator:__ - followed ba a description which can be any text your filesystem supports
- ended with
esMigrationSuffixeswhich is by default.httpand is configurable and case-insensitive.
Elasticsearch-Evolution uses the version for ordering your scripts and enforces strict ordered execution of your scripts. Out-of-Order execution is not supported. Elasticsearch-Evolution interprets the version parts as Integers, so each version part must be between 1 (inclusive) and 2,147,483,647 (inclusive).
Here is and example which indicates the ordering: 1.0.1 < 1.1 < 1.2.1 < (2.0.0 == 2).
In this example version 1.0.1 is the smallest version and is executed first, after that version 1.1, 1.2.1 and in the end 2.
2 is the same as 2.0 or 2.0.0 - so leading zeros will be trimed.
NOTE: Versions with major version 0 are reserved for internal usage, so the smallest version you can define is 1
Elasticsearch-Evolution can be configured to your needs:
- enabled (default=true): Whether to enable or disable Elasticsearch-Evolution.
- locations (default=[classpath:es/migration]): List of locations of migrations scripts. Supported is classpath:some/path and file:/some/path. The location is scanned recursive, but only to a depth of 10. NOTE: all scripts in all locations / subdirectories will be flatted and only the version number will be used to order them.
- encoding (default=UTF-8): Encoding of migration files.
- defaultContentType (default=application/json; charset=UTF-8): This content type will be used as default if no contentType header is specified in the header section of a migration script. If no charset is defined, the
encodingcharset is used. - esMigrationPrefix (default=V): File name prefix for migration files.
- esMigrationSuffixes (default=[.http]): List of file name suffixes for migration files. The suffix is checked case-insensitive.
- placeholderReplacement (default=true): Whether to enable or disable placeholder replacement in migration scripts.
- placeholders (default=[]): Map of placeholders and their replacements to apply to migration scripts.
- placeholderPrefix (default=${): Prefix of placeholders in migration scripts.
- placeholderSuffix (default=}): Suffix of placeholders in migration scripts.
- historyIndex (default=es_evolution): Name of the history index that will be used by Elasticsearch-Evolution. In this index Elasticsearch-Evolution will persist his internal state and tracks which migration script has already been executed.
- historyMaxQuerySize (default=1000): The maximum query size while validating already executed scripts. This query size have to be higher than the total count of your migration scripts.
You can set the above configurations via Spring Boots default configuration way. Just use the prefix spring.elasticsearch.evolution. Here is a example application.properties:
spring.elasticsearch.evolution.locations[0]=classpath:es/migration
spring.elasticsearch.evolution.locations[1]=classpath:es/more_migration_scripts
spring.elasticsearch.evolution.placeholderReplacement=true
spring.elasticsearch.evolution.placeholders.indexname=myIndexReplacement
spring.elasticsearch.evolution.placeholders.docType=_doc
spring.elasticsearch.evolution.placeholders.foo=bar
spring.elasticsearch.evolution.historyIndex=es_evolutionSince spring boot 2.1 AutoConfiguration for Elasticsearchs REST client is provided (see org.springframework.boot.autoconfigure.elasticsearch.rest.RestClientAutoConfiguration).
You can configure the RestHighLevelClient, required for Elasticsearch-Evolution, just like that in your application.properties:
spring.elasticsearch.rest.uris[0]=https://example.com:9200
spring.elasticsearch.rest.username=my-user-name
spring.elasticsearch.rest.password=my-secret-pwElasticsearch-Evolutions just needs a RestHighLevelClient as spring bean.
If you don't have spring boot 2.1 or later or you need a special RestHighLevelClient configuration e.g. to accept self signed certificates or disable hostname validation, you can provide a custom RestHighLevelClient like this:
@Bean
public RestHighLevelClient restHighLevelClient() {
RestClientBuilder builder = RestClient.builder(HttpHost.create("https://localhost:9200"))
.setHttpClientConfigCallback(httpClientBuilder -> {
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("my-user-name", "my-secret-pw"));
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
try {
httpClientBuilder
.setSSLContext(new SSLContextBuilder().loadTrustMaterial(null, TrustAllStrategy.INSTANCE).build())
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
} catch (GeneralSecurityException e) {
throw new IllegalStateException("could not configure http client to accept all certificates", e);
}
return httpClientBuilder;
}
);
return new RestHighLevelClient(builder);
}Maybe you want to provide a customised Initializer for Elasticsearch-Evolution e.g with another Order:
@Bean
public ElasticsearchEvolutionInitializer customElasticsearchEvolutionInitializer(ElasticsearchEvolution elasticsearchEvolution) {
return new ElasticsearchEvolutionInitializer(elasticsearchEvolution) {
@Override
public int getOrder() {
return Ordered.LOWEST_PRECEDENCE;
}
};
}You can set the above configurations via the ElasticsearchEvolutionConfig fluent builder like this:
ElasticsearchEvolution.configure()
.setLocations(Collections.singletonList("classpath:es/migration"))
.setPlaceholderReplacement(true)
.setPlaceholders(Collections.singletonMap("indexname", "myIndexReplacement"))
.setHistoryIndex("es_evolution");- new configuration parameter
historyMaxQuerySize - version updates (spring-boot 2.1.4.RELEASE; elasticsearch 6.7.1)
- version updates (spring-boot 2.1.3.RELEASE; elasticsearch 6.7.0)
- bugfix: support more than 10 migration scripts: now 1000.
- improved logging
- fixed classpath scanning in fat-jars like spring-boot
- initial version