precog/platform

Conversion from raw files to NihDB file format

jdegoes opened this issue · 0 comments

Once the analytics engine can query across raw files (e.g. JSON), as well as NihDB files, we need the ability to convert from raw formats to NihDB, to improve performance.

This needn't be integrated into the query engine (although it could be, as some sort of caching later for queries across raw files), but could instead exist as a separate process which a user could run -- a utility of sorts that would convert data from various formats to NihDB, where queries would run much more efficiently.

When this ticket is done, although users can query across their JSON / CSV files directly, they will be able to achieve increased performance by converting those files into NihDB format.

At minimum, a user will be able to convert files manually by running a command-line process, or through a REST API on the storage side which executes the conversion asynchronously.

The conversion should not delete the original files (users can always do that themselves).