feat: using defaults in schema to populate null value in streams
prakharcode opened this issue · 2 comments
Feature scope
Taps (catalog, state, stream maps, tests, etc.)
Description
Currently in JSON schema, we can define default values but this default is used nowhere. If we can use the stream defaults to process the records with default values it would remove the overhead of using post_process in some of these streams by tap developers.
I was just reading on how jsonschema is trying to solve defaults and was going through #1471
I think this could be implemented by overriding Draft4Validtor like here and then using that as the format validator.
I can give this a try later, wdyt @edgarrmondragon ?
tying back to the discussion here
I was able to figure out that we can use this function to populate default values at target.
Just need to override sink's validator like this and then default values would show up.