Segfault when transferring objects >2gb

Question

Segfault when transferring objects >2gb

Closed this issue 7 years ago · 9 comments

Strange issue, rkdb seems to segfault with a memory not mapped error (bringing down the entire R process) when transferring objects larger than about 2gb. This is using a 64-bit linux machine and the professional version of kdb+/q.

Answer 1 · 2017-08-28T18:59:53.000Z

Can you provide a reproducible example ?

Answer 2 · 2017-08-28T23:49:32.000Z

In q with a set port: ''' t:til 312500000 ''' In R: ''' library(rkdb) # set URL and port variables as appropriate qcon <- rkdb::open_connection(url,port) out <- rkdb::execute(qcon, "t") ''' Crashes my R session with the error: ''' *** caught segfault *** address 0x263a000, cause 'memory not mapped' Segmentation fault '''

…

On Mon, Aug 28, 2017 at 14:59 statquant ***@***.***> wrote: Can you provide a reproducible example ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL2Y2clZRFfalq0JxMwpw-P4gFklB6Kks5scw4qgaJpZM4PD6ve> .

Answer 3 · 2017-09-06T04:21:49.000Z

Is this a reproducible issue for anyone else or just specific to my particular instance?

Answer 4 · 2017-09-06T09:14:37.000Z

Currently, none of the client libraries support message sizes >2GB. This includes c.o which is used by R interface.
You would need to split your data and fetch in chunks.
It doesn't crash for me

> qcon <- rkdb::open_connection()
> out <- rkdb::execute(qcon, "til 312500000")
Error in rkdb::execute(qcon, "til 312500000") : Error from kdb+: `limit

What version of kdb+ are you using and what version of rkdb?

Answer 5 · 2017-09-06T11:37:42.000Z

Sorry should have mentioned that earlier. .z.K - 3.5 .z.k - 2017.04.06 .z.o - 'l64

…

On Wed, Sep 6, 2017 at 05:14 Sergey Vidyuk ***@***.***> wrote: This is known issue. Currently, none of the client libraries support message sizes >2GB. This includes c.o which is used by R interface. You would need to split your data and fetch in chunks. It doesn't crash for me > qcon <- rkdb::open_connection() > out <- rkdb::execute(qcon, "til 312500000") Error in rkdb::execute(qcon, "til 312500000") : Error from kdb+: `limit What version of kdb+ are you using and what version of rkdb? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL2Y2w0iImCUb23d_EFKCsBquJC0Y9Vks5sfmJ9gaJpZM4PD6ve> .

Answer 6 · 2017-09-06T12:53:09.000Z

I cannot reproduce this. Is there any specific command line params you have for kdb+ process? Any errors on kdb side before R crashes?
Could you install latest version of rkdb from github and see if it still happens?

Answer 7 · 2017-09-06T13:41:30.000Z

Interesting, thanks for trying it out. Nothing special besides rlwrap. If I use q 3.2 I get that same limit error. It's just on 3.5 that this segfaults. I will try the latest version.

…

On Wed, Sep 6, 2017 at 08:53 Sergey Vidyuk ***@***.***> wrote: I cannot reproduce this. Is there any specific command line params you have for kdb+ process? Any errors on kdb side before R crashes? Could you install latest version from github and see if it still happens? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL2Y1xkQmWdW9SxrWjH6okW6NgvKpnhks5sfpW2gaJpZM4PD6ve> .

Answer 8 · 2017-09-25T13:47:54.000Z

Did it work for you with 3.5?

Answer 9 · 2017-09-26T03:00:09.000Z

Still segfaults unfortunately. Must be something with my particular environment. Appreciate the help.