dolphindb/DolphinDBPlugin

[New Feature] Development List TO DO

Opened this issue · 2 comments

peeyee commented

To do list for new Contributors

Contributions are welcome, here is a list for new contributors:

No Plugin Content Priority Difficulty Contributer Status Issue/PR
1 clickhouse   low medium      
2 hdf5 add docs about the perfect way of loading data and use cases. medium low      
3 httpClient support for windows high medium      
4 kafka support for kerberos low medium      
5 kafka support for windows high medium      
6 kdb support for using loadTextEx medium medium      
7 kdb support for reading  compressed data of type q IPC (1), snappy (3), L4Z (4) high high      
8 mqtt support for arm low medium      
9 mqtt publish stream engine calculation results directly to the mqtt server medium medium      
10 mysql add docs about return type medium low      
11 mysql support for type blob high high      
12 mysql support for type decimal high high      
13 oracle   high high      
14 parquet support for reading column of  type repeated high high      
15 parquet support for windows high medium      
16 signal support for rfft, irfft high high      
17 xgboost upgrade the library of xgboost to version 1.7 medium medium      
18 zip support for arm low medium      
19 zip support for compressing folders high medium      

Item No.6 kdb+'s tasks: Is it referring to direct file access (similar to loadFile), bypassing q process (similar to loadTable)?

Yes, the way of loadTable/File is totally loading the data into DolphinDB's in-memory table. Thus comes a problem if we do not have enough memory,especially for community users.
So we want to have a method loadTextEx that directly loading data into DolphinDB's DFS table, meanwhile reduce the memory usage, for example, each batch costs 512MB.