Prism: Redshift Spectrum Streaming Loader
Prism is a scalable, fast Redshift Spectrum Streaming Loader. This software is developped at Cookpad in working time.
Prerequisites
- OpenJDK 11
- Tested with Amazon Corretto only, but other distributions like Eclipse Temurin should also work.
Build
% ./gradlew build
Components
- stream/: Prism Stream converts JSONL S3 objects to Parquet objects.
- merge/: Prism Merge merges small Parquet objects into large one.
- batch/: Prism Batch Jobs update Glue Catalog to reflect latest partition info.
- gc/: Prism GC deletes unused merged objects.
Setup
TBD
License
MIT license. See LICENSE file for details.
Authors
- Hidekazu Kobayashi @koba789 (original author)
- Minero Aoki @aamine (current maintainer)