memory corruption
Closed this issue · 7 comments
wonb168 commented
when I use pgscan to query gp table, raise memory corruption
my code
[root@project ~]# python
Python 3.8.13 (default, Aug 16 2022, 12:16:29)
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os #必须加这2行！！！
>>> sys.setdlopenflags(os.RTLD_GLOBAL | os.RTLD_NOW)
>>> import duckdb as dd
>>> print(dd.__version__)
0.5.2-dev1295
>>> con = dd.connect(":memory:", config={"allow_unsigned_extensions": True})
>>> con.execute("INSTALL postgres_scanner;")
<duckdb.DuckDBPyConnection object at 0x7f205ccd2670>
>>> con.execute("LOAD postgres_scanner;")
<duckdb.DuckDBPyConnection object at 0x7f205ccd2670>
>>> gp_str='postgres://密码@192.168.0.161:2345/mdmaster_baosheng_test1'
>>> sql="""create schema tenant_baosheng_edw"""
>>> con.execute(sql)
<duckdb.DuckDBPyConnection object at 0x7f205ccd2670>
>>> sql=f"""create view tenant_baosheng_edw.dim_sku as
... SELECT * FROM postgres_scan('{gp_str}', 'tenant_baosheng_edw', 'dim_sku');"""
>>> con.execute(sql)
<duckdb.DuckDBPyConnection object at 0x7f205ccd2670>
>>> con.execute("select count(*) from tenant_baosheng_edw.dim_sku").fetchall()
Error in `python': malloc(): memory corruption: 0x00007f203400d340
detail:
======= Backtrace: =========
/lib64/libc.so.6(+0x82b36)[0x7f2092fb4b36]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7f2092fb778c]
/lib64/libstdc++.so.6(_Znwm+0x1d)[0x7f20893b218d]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb12VectorBuffer20CreateConstantVectorENS_12PhysicalTypeE+0x2b)[0x7f208a2b4e6b]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb12VectorBuffer20CreateConstantVectorERKNS_11LogicalTypeE+0xd)[0x7f208a2b4f8d]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb6Vector9ReferenceERKNS_5ValueE+0x3c)[0x7f208a2cf09c]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb18ExpressionExecutor7ExecuteERKNS_10ExpressionEPNS_15ExpressionStateEPKNS_15SelectionVectorEmRNS_6VectorE+0xd8)[0x7f208a3032a8]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb18ExpressionExecutor17ExecuteExpressionEmRNS_6VectorE+0x46)[0x7f208a3033c6]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb18ExpressionExecutor7ExecuteEPNS_9DataChunkERS1_+0x4b)[0x7f208a3034cb]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZNK6duckdb18PhysicalProjection7ExecuteERNS_16ExecutionContextERNS_9DataChunkES4_RNS_19GlobalOperatorStateERNS_13OperatorStateE+0x13)[0x7f208a49ada3]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb16PipelineExecutor7ExecuteERNS_9DataChunkES2_m+0x176)[0x7f208aa71426]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb16PipelineExecutor19ExecutePushInternalERNS_9DataChunkEm+0x6b)[0x7f208aa7161b]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb16PipelineExecutor7ExecuteEm+0x5f)[0x7f208aa719ff]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb12PipelineTask11ExecuteTaskENS_17TaskExecutionModeE+0x31)[0x7f208aa78fc1]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb12ExecutorTask7ExecuteENS_17TaskExecutionModeE+0x21)[0x7f208aa705f1]
/opt/rh/rh-python38/root/usr/lib64/python3.8/site-packages/duckdb-0.5.2.dev1295+gd3db91f-py3.8-linux-x86_64.egg/duckdb.cpython-38-x86_64-linux-gnu.so(_ZN6duckdb13TaskScheduler14ExecuteForeverEPSt6atomicIbE+0xb8)[0x7f208aa74c08]
/lib64/libstdc++.so.6(+0xb5330)[0x7f2089408330]
/lib64/libpthread.so.0(+0x7ea5)[0x7f2093a10ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f2093030b0d]
======= Memory map: ========
00400000-00401000 r--p 00000000 08:01 4196452 /opt/rh/rh-python38/root/usr/bin/python3.8
00401000-00402000 r-xp 00001000 08:01 4196452 /opt/rh/rh-python38/root/usr/bin/python3.8
00402000-00403000 r--p 00002000 08:01 4196452 /opt/rh/rh-python38/root/usr/bin/python3.8
00403000-00404000 r--p 00002000 08:01 4196452 /opt/rh/rh-python38/root/usr/bin/python3.8
00404000-00405000 rw-p 00003000 08:01 4196452 /opt/rh/rh-python38/root/usr/bin/python3.8
01480000-022b9000 rw-p 00000000 00:00 0 [heap]
7f2034000000-7f2034021000 rw-p 00000000 00:00 0
7f2034021000-7f2038000000 ---p 00000000 00:00 0
7f2038000000-7f2038021000 rw-p 00000000 00:00 0
7f2038021000-7f203c000000 ---p 00000000 00:00 0
7f203c000000-7f203c021000 rw-p 00000000 00:00 0
7f203c021000-7f2040000000 ---p 00000000 00:00 0
7f2044000000-7f2044021000 rw-p 00000000 00:00 0
7f2044021000-7f2048000000 ---p 00000000 00:00 0
7f204afe6000-7f204afe7000 ---p 00000000 00:00 0
7f204afe7000-7f204b7e7000 rw-p 00000000 00:00 0
7f204b7e7000-7f204b7e8000 ---p 00000000 00:00 0
7f204b7e8000-7f204bfe8000 rw-p 00000000 00:00 0
7f204bfe8000-7f204bfe9000 ---p 00000000 00:00 0
7f204bfe9000-7f204c7e9000 rw-p 00000000 00:00 0
7f204c7e9000-7f204c7ea000 ---p 00000000 00:00 0
7f204c7ea000-7f204cfea000 rw-p 00000000 00:00 0
7f204cfea000-7f204cfeb000 ---p 00000000 00:00 0
7f204cfeb000-7f204d7eb000 rw-p 00000000 00:00 0
7f204d7eb000-7f204d7ec000 ---p 00000000 00:00 0
7f204d7ec000-7f204dfec000 rw-p 00000000 00:00 0
7f204dfec000-7f204dfed000 ---p 00000000 00:00 0
7f204dfed000-7f204e7ed000 rw-p 00000000 00:00 0
7f204e7ed000-7f204e7ee000 ---p 00000000 00:00 0
7f204e7ee000-7f204efee000 rw-p 00000000 00:00 0
7f204efee000-7f204efef000 ---p 00000000 00:00 0
7f204efef000-7f204f7ef000 rw-p 00000000 00:00 0
7f204f7ef000-7f204f7f0000 ---p 00000000 00:00 0
7f204f7f0000-7f204fff0000 rw-p 00000000 00:00 0
7f204fff0000-7f204fff1000 ---p 00000000 00:00 0
7f204fff1000-7f20507f1000 rw-p 00000000 00:00 0
7f20507f1000-7f20507f2000 ---p 00000000 00:00 0
7f20507f2000-7f2050ff2000 rw-p 00000000 00:00 0
7f2050ff2000-7f2050ff3000 ---p 00000000 00:00 0
7f2050ff3000-7f20517f3000 rw-p 00000000 00:00 0
7f20517f3000-7f20517f4000 ---p 00000000 00:00 0
7f20517f4000-7f2051ff4000 rw-p 00000000 00:00 0
7f2051ff4000-7f2051ff5000 ---p 00000000 00:00 0
7f2051ff5000-7f20527f5000 rw-p 00000000 00:00 0
7f20527f5000-7f20527f6000 ---p 00000000 00:00 0
7f20527f6000-7f2052ff6000 rw-p 00000000 00:00 0
7f2052ff6000-7f2052ff7000 ---p 00000000 00:00 0
7f2052ff7000-7f20537f7000 rw-p 00000000 00:00 0
7f20537f7000-7f20537f8000 ---p 00000000 00:00 0
7f20537f8000-7f2053ff8000 rw-p 00000000 00:00 0
7f2053ff8000-7f2053ff9000 ---p 00000000 00:00 0
7f2053ff9000-7f20547f9000 rw-p 00000000 00:00 0
7f20547f9000-7f20547fa000 ---p 00000000 00:00 0
7f20547fa000-7f2054ffa000 rw-p 00000000 00:00 0
7f2054ffa000-7f2054ffb000 ---p 00000000 00:00 0
7f2054ffb000-7f20557fb000 rw-p 00000000 00:00 0
7f20557fb000-7f20557fc000 ---p 00000000 00:00 0
7f20557fc000-7f2055ffc000 rw-p 00000000 00:00 0
7f2055ffc000-7f2055ffd000 ---p 00000000 00:00 0
7f2055ffd000-7f20567fd000 rw-p 00000000 00:00 0
7f20567fd000-7f20567fe000 ---p 00000000 00:00 0
7f20567fe000-7f2056ffe000 rw-p 00000000 00:00 0
7f2056ffe000-7f2056fff000 ---p 00000000 00:00 0
7f2056fff000-7f20577ff000 rw-p 00000000 00:00 0
7f20577ff000-7f2057800000 ---p 00000000 00:00 0
7f2057800000-7f2058022000 rw-p 00000000 00:00 0
7f2058022000-7f205c000000 ---p 00000000 00:00 0
7f205c197000-7f205c200000 r-xp 00000000 08:01 2497030 /root/.duckdb/extensions/d3db91f/linux_amd64_gcc4/postgres_scanner.duckdb_extension
7f205c200000-7f205c400000 ---p 00069000 08:01 2497030 /root/.duckdb/extensions/d3db91f/linux_amd64_gcc4/postgres_scanner.duckdb_extension
7f205c400000-7f205c403000 r--p 00069000 08:01 2497030 /root/.duckdb/extensions/d3db91f/linux_amd64_gcc4/postgres_scanner.duckdb_extension
7f205c403000-7f205c404000 rw-p 0006c000 08:01 2497030 /root/.duckdb/extensions/d3db91f/linux_amd64_gcc4/postgres_scanner.duckdb_extension
7f205c404000-7f205c405000 ---p 00000000 00:00 0
7f205c405000-7f205cc05000 rw-p 00000000 00:00 0
7f205cc05000-7f205cc14000 r--p 00000000 08:01 7080731 /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/pyarrow/_parquet.cpython-38-x86_64-linux-gnu.so
7f205cc14000-7f205cc4e000 r-xp 0000f000 08:01 7080731 /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/pyarrow/_parquet.cpython-38-x86_64-linux-gnu.so
7f205cc4e000-7f205cc5b000 r--p 00049000 08:01 7080731 /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/pyarrow/_parquet.cpython-38-x86_64-linux-gnu.soAborted
Mytherin commented
Do any queries against the Postgres table work, or is it only a specific query that causes the memory corruption? It looks like you are using a custom build and messing with the dlopen flags - it would not surprise me if the problem is related to that.
Mytherin commented
Could you also share which commit hash you are building the DuckDB repository and the Postgres scanner with?
wonb168 commented
Do any queries against the Postgres table work, or is it only a specific query that causes the memory corruption? It looks like you are using a custom build and messing with the dlopen flags - it would not surprise me if the problem is related to that.
because postgres_scanner cannot support array and json cloumns, I edit the postgres_scanner.cpp, and compiled,
my release is git clone -b viking https://github.com/wonb168/postgres_scanner.git, just edit the postgres_scanner.cpp
I compiled more than 10+ times, error: postgres_scanner.duckdb_extension: undefined symbol: _ZN6duckdb28SimpleNamedParameterFunction8ToStringEv, I issued in #44
and then today I find in issue, can set dlopen flags to void it.
but then memory corruption.
Mytherin commented
So you are running your own code? Could you share the modifications that you have made?
wonb168 commented
So you are running your own code? Could you share the modifications that you have made?
limit 2 is OK,
but create view and then query select count(*), error.
my edited code:
else if (pgtypename == "_text")
{
return LogicalType::LIST(LogicalType::VARCHAR);
}else if (pgtypename == "_bpchar")
{
return LogicalType::LIST(LogicalType::VARCHAR);
}else if (pgtypename == "_int4")
{
return LogicalType::LIST(LogicalType::INTEGER);
} else if (pgtypename == "_float4") {
return LogicalType::LIST(LogicalType::FLOAT);
} else if (pgtypename == "_float8") {
return LogicalType::LIST(LogicalType::DOUBLE);
} else if (pgtypename == "_timestamp") {
return LogicalType::LIST(LogicalType::TIMESTAMP);
} else if (pgtypename == "_timestamptz") {
return LogicalType::LIST(LogicalType::TIMESTAMP_TZ);
} else {
throw IOException("Unsupported Postgres type %s", pgtypename);
}
[my postgres_scanner.cpp](https://github.com/wonb168/postgres_scanner/blob/viking/postgres_scanner.cpp)
but, the binary(duckdb cli) all OK, just python have this problem.
hannes commented
Sorry, we cannot help you with your modifications not working
wonb168 commented
Sorry, we cannot help you with your modifications not working
but, postgres_scanner can add this feature in next version, pg contains array、json columns,
now postgres_scanner cannot WORK in most tables.