danzafar/tidyspark

Revitalisation

Opened this issue · 3 comments

With tidytable + tidymodels in R, as well as tidypolars in Python, there is enough justification for 'tidy' to grow more.

It seems that this repo is 2 years old, starting with most of the branches not being pruned.

It doesn't seem to support R 4's native pipes |> either.

And it would be really nice if it's on CRAN.

Please let me know if you don't plan on returning to this repo. Though I lack the knowledge, at least only then I will fork it and try something.

Hey there @exsell-jc , this project was put on pause indefinitely because of issues integrating it into the open-source Spark project. I think there could be more to do here, especially surrounding R4 classes you mentioned to provide a PySpark-esque experience for R developers. What did you have in mind?

I didn't have anything specific in mind except maybe a light wrapper for Spark, similar to tidytable. For now, I just wanted some information.

What are some of the issues with integration?