enhancement : Add a "lob_threshold_kb" parameter to sqlalchemy-ibmi which sets MAXFIELDLEN
openmax opened this issue · 3 comments
Please add new parameter for setting ODBC MAXFIELDLEN ,
need for performance tuning on LOB field ,
and as work around for a bug in implementation of pyodbc ..
extract from a chat with :
Kevin Adler Wed 8:55pm
PyODBC handles SQPutData strangely. It queries SQLGetTypeInfo(SQL_WVCHAR) to get the column size, which it gets back as 16369. It uses this ("char" length) as the max number of bytes to send for SQL_C_WCHAR types for whatever reason. And because this length is over the MAXFIELDLEN size, the column uses a locator instead of inline data. Because of that, the driver will convert each chunk of data as it receives it and send it to the server. This is done to reduce memory use. The problem is that because we got an odd number of bytes of UTF-16 (16369), there's 1 byte left over and the driver discards it (without realizing it). Then when the driver gets the next SQLPutData, we've skipped that dropped byte and so instead of getting eg. 0x41004100... (eg 'AA'), we get 0x00410041... ('䄀...') which is invalid in CCSID 1141.