Go Elasticsearch Connect Couchbase

This repository contains the Go implementation of the Couchbase Elasticsearch Connector.

What is Couchbase Elasticsearch Connector?
Why?
Features
Usage
Configuration
Examples

What is Couchbase Elasticsearch Connector?

Go Elasticsearch Connect Couchbase is a source connector. So it writes Couchbase mutations to Elasticsearch as documents.

Why?

Build a Couchbase Elasticsearch Connector by using Go to reduce resource usage.
Suggesting more configurations so users can make changes to less code.
By presenting the connector as a library, ensuring that users do not clone the code they don't need.

Example

package main

import (
	goelasticsearchconnectcouchbase "github.com/Trendyol/go-elasticsearch-connect-couchbase"
	"github.com/Trendyol/go-elasticsearch-connect-couchbase/couchbase"
	"github.com/Trendyol/go-elasticsearch-connect-couchbase/elasticsearch/document"
)

func mapper(event couchbase.Event) []document.ESActionDocument {
	if event.IsMutated {
		e := document.NewIndexAction(event.Key, event.Value, nil)
		return []document.ESActionDocument{e}
	}
	e := document.NewDeleteAction(event.Key, nil)
	return []document.ESActionDocument{e}
}

func main() {
	connector, err := goelasticsearchconnectcouchbase.NewConnectorBuilder("config.yml").
		SetMapper(mapper).
		Build()
	if err != nil {
		return
	}

	defer connector.Close()
	connector.Start()
}

Custom log structures can be used with the connector

package main

import (
	goelasticsearchconnectcouchbase "github.com/Trendyol/go-elasticsearch-connect-couchbase"
	"github.com/Trendyol/go-elasticsearch-connect-couchbase/couchbase"
	"github.com/Trendyol/go-elasticsearch-connect-couchbase/elasticsearch/document"
	"log"
	"os"
)

func mapper(event couchbase.Event) []document.ESActionDocument {
	if event.IsMutated {
		e := document.NewIndexAction(event.Key, event.Value, nil)
		return []document.ESActionDocument{e}
	}
	e := document.NewDeleteAction(event.Key, nil)
	return []document.ESActionDocument{e}
}

func main() {
	logger := log.New(os.Stdout, "cb2elastic: ", log.Ldate|log.Ltime|log.Llongfile)

	connector, err := goelasticsearchconnectcouchbase.NewConnectorBuilder("config.yml").
		SetMapper(mapper).
		SetLogger(logger).
		SetErrorLogger(logger).
		Build()
	if err != nil {
		return
	}

	defer connector.Close()
	connector.Start()
}

Usage

$ go get github.com/Trendyol/go-elasticsearch-connect-couchbase

Dcp Configuration

Check out on go-dcp-client

Elasticsearch Specific Configuration

Variable	Type	Required	Default
`elasticsearch.collectionIndexMapping`	map[string]string	yes
`elasticsearch.urls`	[]string	yes
`elasticsearch.typeName`	string	no	_doc
`elasticsearch.batchSizeLimit`	int	no	1000
`elasticsearch.batchTickerDuration`	time.Duration	no	10s
`elasticsearch.batchByteSizeLimit`	int	no	10485760
`elasticsearch.batchByteSizeLimit`	int	no	10485760
`elasticsearch.maxConnsPerHost`	int	no	512
`elasticsearch.maxIdleConnDuration`	time.Duration	no	10s
`elasticsearch.compressionEnabled`	boolean	no	false

Examples

example

ramazan/go-elasticsearch-connect-couchbase