Add typed syntax for RDD

Question

Add typed syntax for RDD

chenharryhua opened this issue 4 years ago · 2 comments

Spark Dataset and DataFrame have typed and unsafeTyped syntax to convert them to TypedDataset. it would be convenient to have the typed syntax for RDDs.
Thanks

Answer 1 · 2021-01-06T03:18:18.000Z

@chenharryhua can you provide an example please?

Answer 2 · 2021-01-06T05:39:17.000Z

@imarios, Thanks for asking. I thought we can say: val tds : TypedDataset[Int] = rdd.typed where rdd has type RDD[Int].. I suddenly realized that in order to convert rdd to dataset, we need spark-session so I don't think it is implementable.. your idea? I will close the ticket if you're OK with that.