taosdata/taos-connector-python

taos_fetch_block _v3 is slow

Closed this issue · 4 comments

hadrianl commented

taos-connector-python/taos/cinterface.py

Line 504 in 4ab57be

def taos_fetch_block_v3(result, fields=None, field_count=None):

I just runned some test and found that taos_fetch_block_v3 is pretty slow here, especaily :

  • _crow_binary_to_python_block_v3
  • _crow_nchar_to_python_block_v3
  • taos_is_null

it seems too much cast, too much python loop.it cost at least 80% time to parse the data from c to python. is it any plan to improve this?

sangshuduo commented

Would you like to contribute to improving it? :)

hadrianl commented

I would try to submit a PR when I`m available

hadrianl commented

is it ok to accept the code like this to improve the performance:

  • Cython ,setuptools and compiler are needed
  • Maybe make it optional when fetch data into python object.
  • In my benchmark test, 2~4 times faster than using ctypes
  • 👍1
markcheney1 commented

is it ok to accept the code like this to improve the performance:

  • Cython ,setuptools and compiler are needed
  • Maybe make it optional when fetch data into python object.
  • In my benchmark test, 2~4 times faster than using ctypes

用了pr合入的代码，速度有至少100%的提升，非常了不起。目前我也在探索数据读取的加速，要不然用taospy做回测就要非常久

  • 👍1