PyAirbyte destinations (e.g. `destination-bigquery`) fail with `You must upgrade your platform version to use this connector version.`
aaronsteers opened this issue · 2 comments
Related to: airbytehq/airbyte#43417
The newest specification for destinations will fail if either generationId or minGenerationId is missing in the configured catalog.
Fix would be for PyAirbyte to inject 1 and 1 as static values.
@edgao - Is there any reason why the CDK doesn't do this already, rather than hard fail?
When both values are 1 we get legacy behavior and everything works as-expected, yes?
Now that there are versions in the wild that require this, I'm going to go ahead and do the fix in PyAirbyte, but this seems like an unnecessary and critical breakage for basically all OSS users.
When both values are 1 we get legacy behavior?
I'd recommend using 0 instead (there's some edge case behaviors we're still working on fixing). Also, overwrite syncs won't actually delete old data, because of the next point
rather than hard fail
the generation ID stuff is a pretty significant interface change. In particular - destinations no longer respect destination_sync_mode=OVERWRITE, and instead rely on minimum_generation to trigger that behavior (and destinations have a ton of new logic to support that, i.e. discovering what tables/generations exists in the destination / juggling tables). Building backwards-compatibility into that would have just made it even worse :/
(as for why this wasn't a major version bump in the connector - we didn't want to show the breaking change notice in cloud, b/c it's pointless. And platform doesn't have an easy way to only display that notif in OSS, and the platform x connector compat matrix didn't exist at the time. So it fell to the connector to handle it.)