petl-developers/petl

adoption of bcpandas for mssql batch inserts

vlasvlasvlas opened this issue · 1 comments

vlasvlasvlas commented

Hi, i think would be nice to use the tool bcpandas for the "todb" use,

https://github.com/yehoshuadimarsky/bcpandas

im currently thinking of just invoking bcpandas taking the dataframe and use it to make batch inserts, but would be nice if this can be setup from whitin petl,

thanks!

juarezr commented

Some considerations:

  • petl currently doesn't have abstractions for fast data ingestion like the suggested
  • It would be nice to have an implementation that:
    • Could be transparently used by anybody using todb() function
    • Would seamlessly be applied to any DBApi driver
    • Could improve existing code with no regressions
  • Related:
  • 🚀1