Cannot convert field to unsupported data type StructType([StructField("user_flush_num", ArrayType(IntegerType(), True)),StructField("field2", ArrayType(IntegerType(), True))])

[jia66wei]

ss = SparkSession.builder.appName("123").getOrCreate()
struct = StructType([StructField("user_flush_num", ArrayType(IntegerType(), True)),StructField("field2", ArrayType(IntegerType(), True))])
struct1 = StructType([StructField('f1',StringType(),True), StructField('list',ArrayType(struct))])
data1 = sc.parallelize(["a","b","c"]).map(lambda x: [x,[[[1,2,3],[1,2,3]],[(1,2,3),(1,2,3)]]])
df = ss.createDataFrame(data1,struct1)
df.createOrReplaceTempView("table")
res = ss.sql("select * from table")
print(res.take(10))

out is ok:
[Row(f1='a', list=[Row(user_flush_num=[1, 2, 3], field2=[1, 2, 3]), Row(user_flush_num=[1, 2, 3], field2=[1, 2, 3])]), Row(f1='b', list=[Row(user_flush_num=[1, 2, 3], field2=[1, 2, 3]), Row(user_flush_num=[1, 2, 3], field2=[1, 2, 3])]), Row(f1='c', list=[Row(user_flush_num=[1, 2, 3], field2=[1, 2, 3]), Row(user_flush_num=[1, 2, 3], field2=[1, 2, 3])])]

but, when convert to tfrecord is wrong:

df.write.format("tfrecords").option("recordType", "Example").option(
"codec", "org.apache.hadoop.io.compress.GzipCodec"
).save(output_root + "/train.tfrecord")

wrong info:
Caused by: java.lang.RuntimeException: Cannot convert field to unsupported data type StructType(StructField(item_newsid,IntegerType,true))
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$.org$tensorflow$spark$datasources$tfrecords$serde$DefaultTfRecordRowEncoder$$encodeFeature(DefaultTfRecordRowEncoder.scala:144)
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$$anonfun$encodeExample$1.apply(DefaultTfRecordRowEncoder.scala:64)
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$$anonfun$encodeExample$1.apply(DefaultTfRecordRowEncoder.scala:61)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$.encodeExample(DefaultTfRecordRowEncoder.scala:61)
at org.tensorflow.spark.datasources.tfrecords.DefaultSource$$anonfun$3.apply(DefaultSource.scala:61)
at org.tensorflow.spark.datasources.tfrecords.DefaultSource$$anonfun$3.apply(DefaultSource.scala:58)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:151)
at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:148)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1374)
at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:163)
... 8 more

[jia66wei]

need help, thank you!

[jia66wei]

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto
can help, tfrecord format is limit, example can support:
which is oneof packed BytesList,// FloatList, or Int64List). >