viant/bqtail

Cant change datatype with transforms

ktopcuoglu opened this issue · 1 comments

We have a string field in json files with %99 int data type.
Want to load it as string into transient table, then using safe_cast transform to persist this fields to final table as int64.

tv072: safe_cast(t.tv072 as int64)
tv050: safe_cast(t.tv050 as float64)

(these fields are string in template table.)

"Errors":[{"Message":"Invalid schema update. Field tv072 has changed type from STRING to INTEGER","Reason":"invalid"}]

I think this situation because of both transient and target table use Schema.Template table as template. How can we fix this?

You can define a separate template for a transient table and separate template for dest table
Check https://github.com/viant/bqtail/tree/master/tail#data-ingestion-rules
dest.transient.template
dest.schema.template