StarRocks/starrocks-connector-for-kafka

got "error: The JSON element does not have the requested type" while following the guide to load data

Opened this issue · 0 comments

When I followed the Kafka-connector-starrocks guide to load data to StarRocks, I got the error message below.

ERROR [starrocks-kafka-connector|task-0] catch exception, wait rollback  (com.starrocks.data.load.stream.v2.StreamLoadManagerV2:424)
com.starrocks.data.load.stream.exception.StreamLoadFailException: Transaction prepare failed, db: example_db, table: test-sqlite-jdbc-accounts, label: -0aba9ed9-fd83-41c3-afcf-9be12bbc926f, 
responseBody: {
    "Status": "DATA_QUALITY_ERROR",
    "Message": "Failed to iterate document stream as object. error: The JSON element does not have the requested type.",
    "ErrorURL": "http://10.5.0.3:8040/api/_load_error_log?file=error_log_174479fdeb882113_29549abd35d3bdb2"
}
errorLog: Error: Data quality error: Failed to iterate document stream as object. error: The JSON element does not have the requested type.. Row: parser current location: [{"id":19,"name":"cathy"}]

        at com.starrocks.data.load.stream.TransactionStreamLoader.prepare(TransactionStreamLoader.java:221)
        at com.starrocks.data.load.stream.v2.TransactionTableRegion.commit(TransactionTableRegion.java:247)
        at com.starrocks.data.load.stream.v2.StreamLoadManagerV2.lambda$init$0(StreamLoadManagerV2.java:210)
        at java.lang.Thread.run(Thread.java:750)

I can get the same error message if I run the command below.

curl --location-trusted -u root             \
    -d '[{"id":19,"name":"cathy"}]'                \
    -H "label:test-3"                  \
    -H "format: JSON"    \
    -H "columns:id,name" \
    -H "Expect:100-continue" \
    -XPUT http://localhost:8030/api/example_db/test-sqlite-jdbc-accounts/_stream_load

But if I add a header by -H "strip_outer_array:true", it will succeed.
So I fix the error by adding "sink.properties.strip_outer_array=true" to the properties file of Kafka-connector-starrocks plugin, because the configuration item will cause kafka-connector-starrocks adding "strip_outer_array:true" to http request header when do stream load.
My question is, is it possible to add the "strip_outer_array:true" header automatically by kafka-connector-starrocks, or do not wrap the data with '[ ]'?

BTW, I submitted a PR to modify the guide by adding "sink.properties.strip_outer_array=true" to the properties file.
StarRocks/starrocks#41218