Structured Stremaing schema updates support timeline
Closed this issue · 1 comments
kurtmaile commented
Hi Sam,
You mention that this is not currently supported - is there a timeline for this at all?
Our schemas will almost certainly have attributes added over time post our initial launch (e.g more order attriburtes - these would not be breaking, mostly just addative). Without schema updates through the stream, this makes such schema evolution more difficult in terms of managing streaming jog and also the BQ table management (e.greplaying full historic streams again but with new and overwriting which feels overkill).
Be great to hear how you manage this currently at your company?
Cheers once again!
Kurt
p.s - Im now going away this w/e can we reschedule to next week sometime? Cheers mate
samelamin commented
Hi Kurt,
This isnt a limitation with the connector, but with Spark itself.
A workaround is to enable schema updates then restart the stream, this will
cause any new updates to be put into BigQuery
Give it a try and let me know if you have any further issues
And we can catch up next weekend too!
…On Fri, May 26, 2017 at 2:24 PM, kurtmaile ***@***.***> wrote:
Hi Sam,
You mention that this is not currently supported - is there a timeline for
this at all?
Our schemas will almost certainly have attributes added over time post our
initial launch (e.g more order attriburtes - these would not be breaking,
mostly just addative). Without schema updates through the stream, this
makes such schema evolution more difficult in terms of managing streaming
jog and also the BQ table management (e.greplaying full historic streams
again but with new and overwriting which feels overkill).
Be great to hear how you manage this currently at your company?
Cheers once again!
Kurt
p.s - Im now going away this w/e can we reschedule to next week sometime?
Cheers mate
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#24>, or mute the
thread
<https://github.com/notifications/unsubscribe-auth/AEHLm5khrJsAbta90ap3DN2wKOU1Sga3ks5r9tKSgaJpZM4NnmIw>
.