open-contracting/cove-ocds

Large file options

Opened this issue · 1 comments

jpmckinney commented

At present, large files will exhaust memory, unless open-contracting/lib-cove-ocds#56 is implemented.

Even if that issue is closed, large files will take very long to process, because jsonschema is slow. (lib-cove is also slow, but we can omit the slow parts like aggregates for large files.)

BODS recently added a sample mode, which effectively reduces the size of the file: openownership/cove-bods#79

It also added a background worker, so that large files at least don't timeout: openownership/cove-bods#81