Pyarrow tables instead of dataframe

Question

danthegoodman1 opened this issue a year ago · 2 comments

I tried doing this but inserting grinded to a halt slowly, where with the copying with DFs it was fine. Maybe they were no different tables?

Either way if this is done right then performance should be sigificantly faster for larger batches

Answer 1 · 2023-08-22T00:53:17.000Z

it might actually be the list(map(lambda))...

Answer 2 · 2023-08-22T01:33:13.000Z

It was a copying issue, decided to deep copy during insert to make sure the behavior isn't unexpected, and added deep copy flag