Upgrade confluent.version
Closed this issue · 4 comments
It seems that the ABRIS is using an old version of Confluent (5.3 compared to the latest 7.0)
Line 74 in 60c7681
There have been many new features, e.g. schema reference (https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#referenced-schemas), that have been introduced in new versions. As this library is using and old version, some schema can not be deserialized with ABRIS, e.g. schema having references.
Upgrading confluent.version should be pretty seamless as the API didn't change much, I am happy to open a PR if required.
This is intentional to keep it compatible with Spark.
Please read discussion here: #215 and also here: #165
Abris from version 5 should be able to work with newer confluent versions, but you will probably have a compatibility issues between other libraries in spark, like Avro. But feel free to try, you can override the dependency from Abris to your desired confluent version. I would certainly like to hear from you if you make it work.
Also, new Abris version for Spark 3.2 is planned. Spark 3.2 should work with newer confluent versions much better thanks to upgraded dependencies.
It seems that #248 is indeed upgrading the confluent.version. Just by curiosity, do you have an ETA for the merge & release?
If there are no big issues, it should be released this month.
This new version will be only for Spark 3.2 though. The current version will stay the recommended one for Spark 3.1 and lower.
It makes sense, I assume that due to https://issues.apache.org/jira/browse/SPARK-27733 there are not much to for Spark <= 3.1. Thanks a lot for the help @cerveada , let me close this issue.