BQ Streaming Inserts

Question

BQ Streaming Inserts

Closed this issue 7 years ago · 2 comments

Hi Sam,

You mentioned if more people requested streaming inserts for Structured Steaming youd consider implementing it.

What if I commissioned your time to do this when you are feeling better? It does cost, but at 5 cents per GB, for the benefit of lower latency, cutting out GCS and having duplicates handled gracefully in the sink (bit of a problem for me at the moment), this would be an awesome feature. And noone else is doing it for Spark (that I know of at least).

Just a thought, of course just decline if its of no interest, no problems.

Regardless, hope you are feeling better mate and the eye recovering well!

Take it easy
Kurt

Answer 1 · 2017-07-11T06:17:25.000Z

Hi Kurt Yeah the eye is much better thanks for asking While you are right on that no one is doing it. It is mainly because structured streaming is not in stable release yet. Although this will be with spark 2.2 Sure you can commission my time. I'll dm you my email on Twitter and we can work out details

…

On Mon, 10 Jul 2017 at 20:37, kurtmaile ***@***.***> wrote: Hi Sam, You mentioned if more people requested streaming inserts for Structured Steaming youd consider implementing it. What if I commissioned your time to do this when you are feeling better? It does cost, but at 5 cents per GB, for the benefit of lower latency, cutting out GCS and having duplicates handled gracefully in the sink (bit of a problem for me at the moment), this would be an awesome feature. And noone else is doing it for Spark (that I know of at least). Just a thought, of course just decline if its of no interest, no problems. Regardless, hope you are feeling better mate and the eye recovering well! Take it easy Kurt — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#29>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEHLm2qHY1G09b3x2HbHCMNXcttXC1K9ks5sMn2WgaJpZM4OTT-A> .

Answer 2 · 2017-07-11T07:03:54.000Z

Very cool thanks - great to hear the eye is coming along too. Guess Im jumping the gun a bit using the dbricks runtime 3.0, which has the 2.2 RC - they've GA-ed the kinesis SS source which was a key feature we needed. think the general 2.2 release is imminent anyway. Cool DM me your dets and we can chat. Keep on recovering! Kurt

…

On 11 July 2017 at 07:17, Sam Elamin ***@***.***> wrote: Hi Kurt Yeah the eye is much better thanks for asking While you are right on that no one is doing it. It is mainly because structured streaming is not in stable release yet. Although this will be with spark 2.2 Sure you can commission my time. I'll dm you my email on Twitter and we can work out details On Mon, 10 Jul 2017 at 20:37, kurtmaile ***@***.***> wrote: > Hi Sam, > > You mentioned if more people requested streaming inserts for Structured > Steaming youd consider implementing it. > > What if I commissioned your time to do this when you are feeling better? > It does cost, but at 5 cents per GB, for the benefit of lower latency, > cutting out GCS and having duplicates handled gracefully in the sink (bit > of a problem for me at the moment), this would be an awesome feature. And > noone else is doing it for Spark (that I know of at least). > > Just a thought, of course just decline if its of no interest, no problems. > > Regardless, hope you are feeling better mate and the eye recovering well! > > Take it easy > Kurt > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#29>, or mute the > thread > <https://github.com/notifications/unsubscribe-auth/ AEHLm2qHY1G09b3x2HbHCMNXcttXC1K9ks5sMn2WgaJpZM4OTT-A> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#29 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATIvhdq-2egdqynlmdmaYyomre4ibxzAks5sMxN2gaJpZM4OTT-A> .