xitongsys/parquet-go

Go struct with list of map

acheong08 opened this issue · 0 comments

Functional JSON schema:

{
  "Tag": "name=parquet_go_root, repetitiontype=REQUIRED",
  "Fields": [
    {
      "Tag": "name=abstract, inname=Abstract, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"
    },
    {
      "Tag": "name=embeddings, inname=Embeddings, type=LIST, repetitiontype=OPTIONAL",
      "Fields": [
				{
					"Tag": "name=item, type=FLOAT, repetitiontype=OPTIONAL"
				}
      ]
    },
    {
      "Tag": "name=doi, inname=Doi, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"
    }
  ]
}

Error with struct:

type TestDocument struct {
	Abstract   *string `parquet:"name=abstract, inname=Abstract, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	Embeddings []*struct {
		Item *float32 `parquet:"name=item, type=FLOAT, repetitiontype=OPTIONAL"`
	} `parquet:"name=embeddings, inname=Embeddings, type=LIST, repetitiontype=OPTIONAL"`
	Doi *string `parquet:"name=doi, inname=Doi, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
}

Error:

[NextRowGroup] Column not found: Parquet_go_rootEmbeddingsListElementItem

Schema:

message schema {
        optional binary abstract (STRING);
        optional group embeddings {
                repeated group list {
                        optional float item;
                }
        }
        optional binary doi (STRING);
}