project-samples/go-elasticsearch-import

go-elasticsearch-import

Import flow

Common Architectures

Layer Architecture

• Popular for web development

Hexagonal Architecture

• Suitable for Import Flow

Based on the flow, there are 4 main components (4 main ports):

Reader

Reader Adapter Sample: File Reader. We provide 2 file reader adapters:

• Delimiter (CSV format) File Reader
• Fix Length File Reader

Validator

• Validator Adapter Sample: Schema Validator
• We provide the Schema validator based on GOLANG Tags

Transformer

We provide 2 transformer adapters

• Delimiter Transformer (CSV)
• Fix Length Transformer

Writer

We provide many writer adapters:

• SQL Writer: to insert or update data

• SQL Inserter: to insert data

• SQL Updater: to update data

• SQL Stream Writer: to insert or update data. When you write data, it keeps the data in the buffer, it does not write data. It just writes data when flush.

• SQL Inserter: to insert data. When you write data, it keeps the data in the buffer, it does not write data. It just writes data when flush. Especially, we build 1 single SQL statement to improve the performance.

• SQL Updater: to update data. When you write data, it keeps the data in the buffer, it does not write data. It just writes data when flush.

• ElasticSearch Writer: to insert or update data

• ElasticSearch Inserter: to insert data

• ElasticSearch Updater: to update data

• ElasticSearch Stream Writer: to insert or update data. When you write data, it keeps the data in the buffer, it does not write data. It just writes data when flush.

• ElasticSearch Inserter: to insert data. When you write data, it keeps the data in the buffer, it does not write data. It just writes data when flush.

• ElasticSearch Updater: to update data. When you write data, it keeps the data in the buffer, it does not write data. It just writes data when flush.