How to specify schema while reading file from S3 bucket

Question

How to specify schema while reading file from S3 bucket

Opened this issue 4 years ago · 2 comments

How to specify schema while reading file from S3 bucket? I have S3 object which contains one of the column as DATE, when I read without schema, I get the DATE as integer value?

{
VERSION_KEY: 2,
VERSION_DESCRIPTION: 'XXX',
EFFECTIVE_DATE: '43200000000000'
}

I am using the method
const reader = await parquet.ParquetReader.openS3(client, params);

Answer 1 · 2021-11-12T20:54:45.000Z

@dkommineni ... Did you find a workaround for this? The number isn't a timestamp so unless I am missing something, there is no way to turn the number into a date.

Answer 2 · 2021-11-15T15:26:57.000Z

Just in case someone else comes upon this .. my issue stemmed from the field being an int96 which is no longer officially part of the Parquet spec and so, not supported by this library which makes sense.