bxparks/bigquery-schema-generator

support sending an existing schema to deduce_schema

lior-lemonade opened this issue · 2 comments

support sending an existing schema to deduce schema so we can merge an existing BigQuery schema with new rows in file.
Something like:

def deduce_schema(self, file, schema_map =None):
if schema_map is None:
schema_map = OrderedDict()

Hi, Can you provide more details about where and how you would use this feature? It sounds like you want to incrementally build the bigquery schema, based on new rows. But it's not clear to me how this would be useful, because (as far as I am aware) BigQuery does not support updating the schema of a table (except for the most trivial cases, e.g. required to nullable). So if the new rows changed the table schema, you'd have to reimport the entire dataset into a new version of the table (using the new schema).

Just a quick note: In the absence of further comments, I will be closing this in my next house cleaning sweep.