Python or Java support?

[linbrian]

Hello,

I was wondering if the HyperLogLog functions can be used in Java or Python (through Pyspark). I would assume so for Java. If there is support, I was wondering if examples can be posted on the Wiki.

Thank you!

[cozos]

Hi,

Based on these:

https://github.com/swoop-inc/spark-alchemy/wiki/Spark-Native-Functions

To use the functions from SparkSQL, you have to register them by calling the equivalent of HLLFunctionRegistration.register(spark)

And https://github.com/swoop-inc/spark-alchemy/blob/master/alchemy/src/main/scala/com/swoop/alchemy/spark/expressions/NativeFunctionRegistration.scala#L20-L22

It seems like you can use all the functions here through SparkSQL:
https://github.com/swoop-inc/spark-alchemy/blob/master/alchemy/src/main/scala/com/swoop/alchemy/spark/expressions/hll/HLLFunctionRegistration.scala#L9-L16

[ssimeonov]

@cozos that's right and naive DSL support in other languages can be built by wrapping functions.expr(<SparkSQL expression>).

@linbrian native DSL support for other languages requires a bit more work but there are lots of examples in Spark for how to do it. PRs welcome. :)

[ssimeonov]

Here is how to register the Spark native functions for HLL from Python: #9 (comment)

Java can use the Scala registration code directly.