eBay/akutan

Log service

Opened this issue · 7 comments

superfell commented

There should be a server implementation of the logspec service. Beam was moving from using Kafka to using this abstract log service definition for its log. The Kafka client needs some work for production usage.

Deleted user commented

I am very interested in the abstraction because I would like to support NATS instead of Kafka.
Just raising this in case others might be thinking to do it.

superfell commented

There's 2 abstraction points for the log that can be used.

  • The gRPC log spec interface, a log service can implement this, and Beam can use this directly.
  • Or you can implement the BeamLog go interface (which is how Kafka is currently integrated)
superfell commented

NATS looks interesting, the NATS streaming service looks it could work. However one thing i've seen is that Publishing a message doesn't return its sequence number in the Log. This is problematic as the writers to the log in the API tier need to know the sequence / log index number writes ended up at, for example, transactions are identified by their log seq #.
The guid's that the AsyncPublish call expose aren't included in the messages the subscriber sees, so you couldn't even have it use a private subscriber to correlate on the guid to then get the seq#.
Its possible i missed something, but looks like it wouldn't be possible to use the NATS streaming service with its current client API as a log for Beam.

superfell commented

Facebook's LogDevice looks like it'd be a good fit for the log service.

joeblew99 commented

@superfell
NATS Streaming has those things and i agree they are needed for your use case.
Its explained here:
https://nats.io/documentation/streaming/nats-streaming-protocol/#PUBACK

Also the GO team are making a Enterprise framework and PUBSUB is in it.
https://github.com/google/go-cloud/tree/master/pubsub

joeblew99 commented

AH its does not support ACK and NACK.
https://github.com/google/go-cloud/blob/master/pubsub/natspubsub/nats.go#L395
SO basically they have NATS but not NATS Streaming support.

superfell commented

Its good its in the protocol, its unfortunate that its not reflected in the go api i looked at
https://github.com/nats-io/go-nats-streaming