use DB to mediate access to raw records

Question

use DB to mediate access to raw records

Closed this issue 4 years ago · 2 comments

load raw records into a DB table (e.g. as CLOBs)
feed PatientFlatReader from a Reader that gets data from the DB table

The "data lake" approach here was somewhat half-hearted: code for reading the NAACCR file had to run on the host with the NAACCR file, since we didn't deploy a distributed filesystem such as HDFS. This would treat the DB somewhat like a distributed filesystem. Using the DB to mediate access to the DB is the norm in HERON development for a decade or so in any case.

p.s. this is the motivation for the loadRaw experiment.

Answer 1 · 2020-05-01T16:29:09.000Z

This is the norm as of the electric release cc28f83

Answer 2 · 2020-07-08T13:36:29.000Z

Performance of using PatientFlatReader from CLOBs to discrete columns was poor: >2.5hrs for ~100k tumor records.

Going straight from the flat file to JDBC gets it down to ~6min in 912ee3a. (a critical bug was fixed in d321e28)