Converts value NA in concept to NULL which results in an error
bmosseveld opened this issue · 4 comments
While loading the vocabulary it crashed on a NOT NULL constraint on concept_name.
The record where it occurs is:
36311145 NA Meas Value LOINC Answer S LA12334-1 19700101 20991231
which is converted by ETL-Synthea to:
36311145,NULL,'Meas Value','LOINC','Answer','S','LA12334-1','1970-01-01 +01','2099-12-31 +01',NULL
The value 'NA' should not be read as NULL.
The replacement occurs here:
ETL-Synthea/R/LoadVocabFromCsv.r
Line 60 in 32c22e1
@bmosseveld Hi Mees, I think a missing concept_name suggests a problem in the data. Removing the conversion of NA to NULL just causes your db to store an empty string, '', in its place. This applies to @katy-sadowski in her comment here as well.
@clairblacketer Clair, I'm thinking that a row in vocab data that is missing something as fundamental as the concept_name should either raise an error (current behavior) or be removed. What's the official position here, from the pov of vocab and cdm?
Thanks for the reply @AnthonyMolinaro ! I should have provided I bit more context in my reply on the other thread - the offending row turned out to be a LOINC code for which the concept name is actually "NA". So the type_convert
function I linked above is converting this string "NA" to an actual NA (NULL).
@katy-sadowski Hey Katy, sorry for the delay! I'm working on this now and will have the fix in today or Monday.