Use arrow + shared memory to pass data between Go and Python

Question

Use arrow + shared memory to pass data between Go and Python

Opened this issue 6 years ago · 5 comments

We'd like to use Arrow + shared memory to pass data between Go & Python. We assume client and server will be on the same machine.

Answer 1 · 2019-02-28T15:09:36.000Z

Talking to the arrow team, they confirm Go is lacking arrow IPC support.

Answer 2 · 2019-02-28T15:10:50.000Z

I've started a small project to test this approach, it is here

Answer 3 · 2019-03-17T08:16:43.000Z

After trying for 1 more day and going over the Go arrow bindings, I've reached the conclusion that the current state of Go arrow does not provide us what we want and will require a lot of work to get there.

IMO the faster (devleopment wise) approach will be to use the C/C++ arrow from Go. It's much more mature and since Python uses the same code we can easilty share memory, probably using the plasma store

Answer 4 · 2019-03-17T09:15:00.000Z

The other option is to use IPC which uses flatbuffers implementation. We can encode/decode to the arrow flatbuffers representation.

Answer 5 · 2019-03-17T16:46:31.000Z

Started minimal implemtation of using C++ arrow from Go here