gbif/pipelines

Enable Snappy compression

Opened this issue · 1 comments

fmendezh commented
  1. Evaluate if Snappy is still a better options to write Avro and Parquet file.
  2. Investigate how to enable Snappy in Spark 3 jobs and how to use it in https://github.com/gbif/occurrence/tree/dev/occurrence-table-build-spark
zaultooz commented

The stackable team is working on enabling the Snappy compression by default which should be included in a minor release.