Aiven's OpenSearch® Connector for Apache Kafka®

This repository includes Aiven's OpenSearch Apache Kafka® Connector for Apache Kafka®.

The project originates from Aiven's elasticsearch-connector-for-apache-kafka. The code was forked and all classes were renamed.

Documentation

How to install

Connector plugins are packaged in zip/tar/tar.gz format to be released
Users download plugins from GitHub releases or build binaries from source
Users place connector plugins on Connect worker instances and add them via configuration
Start creating connectors using installed plugins

Download binaries

Binaries are included on every release as zip/tar files: https://github.com/aiven/opensearch-connector-for-apache-kafka/releases/latest

Build from Source

Execute gradle task to build binaries:

./gradlew installDist
# or ./gradlew assembleDist to package binaries

This produces an output on build/install directory with the plugin binaries to add into Connect cluster.

Add plugin to Connect worker

Place unpacked binaries into a directory on each Connect worker node, e.g. /kafka-connect-plugins.

In this case, place opensearch-connector-for-kafka into /kafka-connect-plugins:

/kafka-connect-plugins
└── opensearch-connector-for-apache-kafka

Then, on each connect worker configuration make sure to add /kafka-connect-plugins to the plugin.path configuration:

plugin.path=/kafka-connect-plugins

Validate Connector plugin installation

Once placed on each worker node, start the workers and check the plugins installed and check the plugin (with the correct version) is included:

# Go to connector rest api
curl http://localhost:8083/connector-plugins | jq .

[
  ...
  {
    "class": "io.aiven.kafka.connect.opensearch.OpensearchSinkConnector",
    "type": "sink",
    "version": "3.0.0"
  },
  ...
]

Connector Configuration

OpenSearch® Sink Connector Configuration Options

Contribute

Source Code

Issue Tracker

License

The project is licensed under the Apache 2 license. See LICENSE.

Trademark

Apache Kafka, Apache Kafka Connect are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

OpenSearch is a trademark and property of its respective owners. All product and service names used in this website are for identification purposes only and do not imply endorsement.

rafis/opensearch-connector-for-apache-kafka