Structured Stremaing schema updates support timeline

Question

Structured Stremaing schema updates support timeline

Closed this issue 7 years ago · 1 comments

Hi Sam,

You mention that this is not currently supported - is there a timeline for this at all?

Our schemas will almost certainly have attributes added over time post our initial launch (e.g more order attriburtes - these would not be breaking, mostly just addative). Without schema updates through the stream, this makes such schema evolution more difficult in terms of managing streaming jog and also the BQ table management (e.greplaying full historic streams again but with new and overwriting which feels overkill).

Be great to hear how you manage this currently at your company?

Cheers once again!
Kurt

p.s - Im now going away this w/e can we reschedule to next week sometime? Cheers mate

Answer 1 · 2017-05-26T13:37:34.000Z

Hi Kurt, This isnt a limitation with the connector, but with Spark itself. A workaround is to enable schema updates then restart the stream, this will cause any new updates to be put into BigQuery Give it a try and let me know if you have any further issues And we can catch up next weekend too!

…

On Fri, May 26, 2017 at 2:24 PM, kurtmaile ***@***.***> wrote: Hi Sam, You mention that this is not currently supported - is there a timeline for this at all? Our schemas will almost certainly have attributes added over time post our initial launch (e.g more order attriburtes - these would not be breaking, mostly just addative). Without schema updates through the stream, this makes such schema evolution more difficult in terms of managing streaming jog and also the BQ table management (e.greplaying full historic streams again but with new and overwriting which feels overkill). Be great to hear how you manage this currently at your company? Cheers once again! Kurt p.s - Im now going away this w/e can we reschedule to next week sometime? Cheers mate — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#24>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEHLm5khrJsAbta90ap3DN2wKOU1Sga3ks5r9tKSgaJpZM4NnmIw> .