jorgecarleitao/arrowdantic

Exporting to numpy & cloud file systems

benjaminrwilson opened this issue · 1 comments

benjaminrwilson commented

Hi @jorgecarleitao,

Thanks for putting this library together -- it looks awesome! I had a few quick questions.

  1. What is the easiest way to convert to / from numpy using arrowdantic?
  2. Do you have any recommendations for reading Arrow files from cloud storage (e.g., s3 or gcs) that are backed by rust with python bindings?
jorgecarleitao commented

Hey @benjaminrwilson , sorry for the late reply, only now noticed this issue.

  • Currently we only support from/to iterators, so it would need to be through that. This is not very efficient, though.
  • The interfaces to read files support a Python file descriptor. So, use a Python lib to get that interface for the cloud storage (e.g. boto3's get_object) and pass it to the file reader.