greenplum-db/pxf

Does GreenPlum with PXF support avro data with schema evolution

vikramnshinde opened this issue · 2 comments

vikramnshinde commented

We have user data (avro files) validated and ingested into HDFS using Schema Registry(data keep on evolving) and using GreenPlum with PXF to access HDFS data. Created one external table and trying to query the HDFS data but getting error as,

warehouse=# select * from user;
ERROR:  Record has 151 fields but the schema size is 152  (seg1 slice1 192.168.1.17:6001 pid=6582)
CONTEXT:  External table user
warehouse=#

user HDFS files are ingested using different schema versions, and GreenPlum external table has been created with fields from all the schema versions.

frankgh commented

@vikramnshinde schema evolution in Avro is not supported, we'll be looking into supporting schema evolution for Avro

vikramnshinde commented

@vikramnshinde schema evolution in Avro is not supported, we'll be looking into supporting schema evolution for Avro

thanks @frankgh for the information, and really appreciate your help!