bug: schema comparison before adding metadata columns
Closed this issue · 0 comments
pnadolny13 commented
#1777 was resolved by #1778 but theres a very similar bug in _process_schema_message in the target where the comparison always returns false when SDC metadata columns are included.
My shopify stream has a bunch of schema messages that cause it to constantly log the fact that it needs to reinitialize a sink because the schema changed. When I added a breakpoint I found that the only difference is the SDC properties.
2023-09-15 16:09:49,858 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:09:50,537 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:09:50,537 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:09:50,537 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:09:50,537 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:09:53,004 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:09:53,004 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:09:53,004 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:09:53,004 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:09:54,800 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:09:54,800 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:09:54,800 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:09:54,800 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:09:56,622 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:09:56,622 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:09:56,622 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:09:56,622 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:09:58,445 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:09:58,445 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:09:58,445 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:09:58,445 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:10:00,275 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:10:00,275 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:10:00,275 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:10:00,275 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:10:02,076 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:10:02,076 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:10:02,076 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:10:02,076 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:10:03,926 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:10:03,926 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:10:03,926 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:10:03,926 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:10:05,793 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:10:05,794 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:10:05,794 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:10:05,794 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:10:07,671 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:10:07,672 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:10:07,672 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:10:07,672 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...
2023-09-15 16:10:09,516 | INFO | target-postgres | Schema has changed for stream 'shopify_transactions'. Mapping definitions will be reset.
2023-09-15 16:10:09,516 | INFO | target-postgres | Schema or key properties for 'shopify_transactions' stream have changed. Initializing a new 'shopify_transactions' sink...
2023-09-15 16:10:09,516 | INFO | target-postgres | Initializing 'target-postgres' target sink...
2023-09-15 16:10:09,516 | INFO | target-postgres | Initializing target sink for stream 'shopify_transactions'...