rcongiu/Hive-JSON-Serde

Unimplemented error from map in array of structs

dougscher opened this issue · 4 comments

dougscher commented

The following hive schema throws an unimplemented exception:
ALTER TABLE TABLE CHANGE ctx ctx
struct<
action:struct<fld1:string, arr1:array<struct<fld2:string, fld3:string, map1:map<string, string>>>>

If you remove map1, it works. Not sure if this is a serde issue or a hive issue.

dougscher commented

Can you please look into this. The workaround is nasty.
It looks like there is a failure in handling a map in an array in a struct.
Thanks,
Doug

rcongiu commented

So, for starters, it is not a SerDe issue, the SerDe can handle nested Maps just fine, it's Hive actually that does not allow to change changing the type of a column in an incompatible way (see https://issues.apache.org/jira/browse/HIVE-4409).
Second, please next time you submit an issue, please try to supply as much info as possible to reproduce the issue, including:

  • CREATE TABLE statement
  • data sample
  • stack trace of the error you're getting.
dougscher commented

This failure was occurring on empty tables that had just been populated.
I am aware that old data can trip up an altered schema...

Can you please verify that this json is compatible with the serde:
{
action: {
fld1: 'fld1',
arr1: [{ fld2: 'fld2', fld3: 'fld3', map1: { m1: 'm1', m2: 'm2' } }]
}
}

Assuming a schema of:
struct<action:struct<fld1:string, arr1:array<struct<fld2:string, fld3:string, map1:map<string, string>>>>

Thanks,
Doug

rcongiu commented

It doesn't matter if it does have data or not, hive won't allow an incompatible schema change. Data is compatible with the serde. Create the table with the right schema at the beginning and load the data into it , you'll see it works.