gbif/pipelines

Fragmenter process running out of memory

Closed this issue · 1 comments

After some days fragmenting eBird, the fragmenter runs out of memory.

Using Spark for big (>20m) datasets. Deployed to PROD