[Feature Request] Support lazy load/zero copy when using shared memory

[vtk9]

Describe the enhancement requested

Use case: there is an arrow table in shared memory. It was written using Python's Arrow RecordBatchStreamWriter (although even if it was written using RecordBatchFileWriter, I believe the same problem would exist due to the current state of the Go Arrow implementation). We would like to load it lazily and stream it directly to an arrow flight client. We would do so by opening up the file in process memory (lazily), then we would first aggregate the list of records (lazily) and then read the contents of the records for the first time when streaming the data to the client.

via

func (s *MyServer) DoGet(
	ticket *flight.Ticket,
	server flight.FlightService_DoGetServer,
) error {
    ...
	w := flight.NewRecordWriter(server, writerOpts...)
	for _, r := range s.recordsInTable {
		if err = w.Write(r); err != nil {
			return err
		}
	}

Ideally, up until the records sent, no data would be actually copied/ready. However, this doesn't appear to be possible at the moment since there is a copy done when processing the records https://github.com/apache/arrow-go/blob/main/arrow/ipc/message.go#L217 instead or referencing the already allocated/used shared memory. Instead, there might need to be a new MessageReader implementation where this copy is avoided.

Thank you!

Component(s)

Other

[zeroshade]

@vtk9 Could you provide an example of the code for writing to shared memory and getting the address / access to the memory in Go?

I want to make sure that whatever solution I come up with will work for your use case.