Avro union type issue with Schema Registry

[calvinlfer]

Hey there,

I accidentally posted this issue in the vulcan project instead of this one so I'm cross-posting the issue here for greater visibility. The gist of it is that you cannot use sealed traits (union types) as top-level subjects when publishing to Kafka and I think this is because of Confluent's underlying serialization mechanism not having proper support for union types. I managed to work around this by using a record containing a schema but I was wondering if this is something we could improve upon since Confluent's Schema Registry does seem to correctly support union types but I believe its the Java Serializer from Confluent that has this limitation.

Thank you so much for all your hard work and this amazing library

[bplommer]

You're right, this is a limitation we inherit from the Confluent Java serializer. An alternative workaround is to manually register the union schema, perhaps on application startup - #687 will provide support for this.

There's also an issue (#506) to reimplement this functionality ourselves, which would allow us to remove this limitation.

[calvinlfer]

Hey @bplommer is there any interest in writing our own Avro serializer? I have done some very similar work for serializers/deserializers for JSON schema, I ended up reverse-engineering the Kafka serializer to come up with the implementation, and thanks to Scala did some additional checks. I think we can use the same approach for Avro - I think that the changes we would need to make would be the Avro validation and serialization/deserialization which we should be able to use once Vulcan has the ability to convert to binary and delegate to the underlying Confluent Schema Registry Client (or the new http4s schema registry client😄) to run the checks and do schema registration. I think this would be immensely useful since we'll get support for sealed traits mapping nicely to Union types

[bplommer]

Hi @calvinlfer, sorry for the very slow reply! Reimplementing Avro serialisers is definitely something I'd like to do - I have a (kind of abandoned) work-in-progress PR for Vulcan at fd4s/vulcan#289 so if you feel like collaborating on it that would be great!