zgbjgg/jun

Big opaque terms

Closed this issue · 1 comments

Check if can change opaque term representing dataframes or series to a single key in the python process, so dataframe will live only in python side.

Running large datasets maybe will cause invalid_protocol_data but I'm not sure about, however this will be a great improve.

The error invalid_protocol_data is thrown when using the same worker for different process, for example, sending a dataframe to get max and then sending other dataframe to get min, since the port on erlang side will colapse decoding data, however this will be avoid with a mechanism of pooling such as poolboy, checkout and checkin the worker 👍