OHDSI/ETL-Synthea

Converts value NA in concept to NULL which results in an error

bmosseveld opened this issue · 4 comments

While loading the vocabulary it crashed on a NOT NULL constraint on concept_name.
The record where it occurs is:

36311145 NA Meas Value LOINC Answer S LA12334-1 19700101 20991231

which is converted by ETL-Synthea to:

36311145,NULL,'Meas Value','LOINC','Answer','S','LA12334-1','1970-01-01 +01','2099-12-31 +01',NULL

The value 'NA' should not be read as NULL.

The replacement occurs here:

na = c(NA, "")) %>%

@bmosseveld Hi Mees, I think a missing concept_name suggests a problem in the data. Removing the conversion of NA to NULL just causes your db to store an empty string, '', in its place. This applies to @katy-sadowski in her comment here as well.
@clairblacketer Clair, I'm thinking that a row in vocab data that is missing something as fundamental as the concept_name should either raise an error (current behavior) or be removed. What's the official position here, from the pov of vocab and cdm?

Thanks for the reply @AnthonyMolinaro ! I should have provided I bit more context in my reply on the other thread - the offending row turned out to be a LOINC code for which the concept name is actually "NA". So the type_convert function I linked above is converting this string "NA" to an actual NA (NULL).

@katy-sadowski Hey Katy, sorry for the delay! I'm working on this now and will have the fix in today or Monday.