Schema introspection method
Closed this issue · 0 comments
danthegoodman1 commented
Function to just get the schema of incoming dicts so that ingestion can compare known schema before inserting to check for conflict.
For example if a single data source maps to a single partition in kafka, then you know one host will ever insert at a time, this means they can cache the known schema in memory, and verify schema for incoming batches before inserting (and not breaking the table). Then they can quarantine offending rows kind of like tinybird does.