Unable to read complex data types(e.g. Map, Struct) after upgrading to latest (0.10.0) version
devendrasr opened this issue · 12 comments
The extension is able to read complex data types using duckdb 0.9.0 -
We could not read complex data types after upgrading to duckdb 0.10.0 -
The database is throwing below error -
Error: IO Error: Invalid field found while parsing field: type
Yes, i ran into #39 after adding a temp workaround for this one.
I had to disable schema parsing and supplying to parquet in order to make things work for the time being.
@devendrasr Can you provide more details on how you overcame the issue? How did you disable schema parsing? Thanks
@harel-e i made few changes like this - devendrasr@d9e5d72
@devendrasr will PR#43 that you created solve this issue? Thank you
It just allows you to skip schema inference and fallback to parquet extension to infer schema. In this way we are able to read complex data types via iceberg extension.
@devendrasr - I'd like to test your changes locally, especially with issue #39. Is it possible for you to provide a binary since I'm having issues building DuckDB locally? thanks.
@harel-e You can try them out using -
docker run --rm --name=duckdb -it devendrasr/duckdb:0.0.1 bash
@devendrasr - thank you for providing an image for testing.
I tried the scenario described here: #39 (comment)
Unfortunately, it still fails with:
D SELECT * FROM iceberg_scan('data/iceberg/lineitem_iceberg', allow_moved_paths = true);
Error: IO Error: Invalid field found while parsing field: required
I understand you were not trying to address issue #39. I thought they were somewhat related.
Thanks again.
My bad, I forgot to add skip_schema_inference = true.
I tested it again with both lineitem data and my own dataset (minio/nessie) and it works !
Thank you very much !
I'll wait for the PR to be merged.
The issue has been resolved in pull request - #45
The request is now merged in trunk. Closing this one.