UNKNOWN, srcAddress /0.0.0.0:0
dacrespi opened this issue · 33 comments
dacrespi commented
I'm attempting to run crail-spark.
I'm set up as a container running spark with workers, and attempting to just access crail store.
Running either crail fs -ls -R / or say terasort, I end up at the same error.
INFO disni: got event type + UNKNOWN, srcAddress /0.0.0.0:0, dstAddress /192.168.3.100:4420
I've set disni in crail to log DEBUG but I don't get any additional info.
It appears that DiSNI is attempting to set up a QP but is unable to determine the local rNIC's adddress? I do have the container setup as a bridged network, rather than a host network, which I'm guessing could be the issue? I do see all the RDMA nodes from the container however. I tried to use a host network but then got Spark errors because it can't have unique hostnames when attached to the host network.
$ ibv_devices
device node GUID
------ ----------------
i40iw0 0cc47afc00ed0000
mlx5_2 98039b0300989ab6
mlx5_0 98039b0300989b0e
i40iw1 0cc47afc00ec0000
mlx5_3 98039b0300989ab7
mlx5_1 98039b0300989b0f
Snippet of the console output prior to the hang:
19/06/12 08:49:10 INFO crail: CrailHadoopFileSystem construction
19/06/12 08:49:10 INFO crail: creating singleton crail file system
19/06/12 08:49:10 INFO crail: crail.version 3101
19/06/12 08:49:10 INFO crail: crail.directorydepth 16
19/06/12 08:49:10 INFO crail: crail.tokenexpiration 10
19/06/12 08:49:10 INFO crail: crail.blocksize 1048576
19/06/12 08:49:10 INFO crail: crail.cachelimit 0
19/06/12 08:49:10 INFO crail: crail.cachepath /dev/hugepages/cache
19/06/12 08:49:10 INFO crail: crail.user crail
19/06/12 08:49:10 INFO crail: crail.shadowreplication 1
19/06/12 08:49:10 INFO crail: crail.debug true
19/06/12 08:49:10 INFO crail: crail.statistics true
19/06/12 08:49:10 INFO crail: crail.rpctimeout 1000
19/06/12 08:49:10 INFO crail: crail.datatimeout 1000
19/06/12 08:49:10 INFO crail: crail.buffersize 1048576
19/06/12 08:49:10 INFO crail: crail.slicesize 524288
19/06/12 08:49:10 INFO crail: crail.singleton true
19/06/12 08:49:10 INFO crail: crail.regionsize 1073741824
19/06/12 08:49:10 INFO crail: crail.directoryrecord 512
19/06/12 08:49:10 INFO crail: crail.directoryrandomize true
19/06/12 08:49:10 INFO crail: crail.cacheimpl org.apache.crail.memory.MappedBufferCache
19/06/12 08:49:10 INFO crail: crail.locationmap
19/06/12 08:49:10 INFO crail: crail.namenode.address crail://192.168.1.164:9060
19/06/12 08:49:10 INFO crail: crail.namenode.blockselection roundrobin
19/06/12 08:49:10 INFO crail: crail.namenode.fileblocks 16
19/06/12 08:49:10 INFO crail: crail.namenode.rpctype org.apache.crail.namenode.rpc.tcp.TcpNameNode
19/06/12 08:49:10 INFO crail: crail.namenode.log
19/06/12 08:49:10 INFO crail: crail.storage.types org.apache.crail.storage.nvmf.NvmfStorageTier
19/06/12 08:49:10 INFO crail: crail.storage.classes 2
19/06/12 08:49:10 INFO crail: crail.storage.rootclass 0
19/06/12 08:49:10 INFO crail: crail.storage.keepalive 2
19/06/12 08:49:10 INFO crail: buffer cache, allocationCount 0, bufferCount 1024
19/06/12 08:49:10 INFO crail: Initialize Nvmf storage client
19/06/12 08:49:10 INFO crail: crail.storage.nvmf.ip 192.168.3.100
19/06/12 08:49:10 INFO crail: crail.storage.nvmf.port 4420
19/06/12 08:49:10 INFO crail: crail.storage.nvmf.nqn nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/12 08:49:10 INFO crail: crail.storage.nvmf.hostnqn nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
19/06/12 08:49:10 INFO crail: crail.storage.nvmf.allocationsize 1073741824
19/06/12 08:49:10 INFO crail: crail.storage.nvmf.queueSize 64
19/06/12 08:49:10 INFO narpc: new NaRPC server group v1.0, queueDepth 32, messageSize 512, nodealy true
19/06/12 08:49:10 INFO crail: crail.namenode.tcp.queueDepth 32
19/06/12 08:49:10 INFO crail: crail.namenode.tcp.messageSize 512
19/06/12 08:49:10 INFO crail: crail.namenode.tcp.cores 1
19/06/12 08:49:10 INFO crail: connected to namenode(s) /192.168.1.164:9060
19/06/12 08:49:10 INFO crail: CrailHadoopFileSystem fs initialization done..
19/06/12 08:49:10 INFO crail: lookupDirectory: path /
19/06/12 08:49:10 INFO crail: lookup: name /, success, fd 0
19/06/12 08:49:10 INFO crail: lookupDirectory: path /
19/06/12 08:49:10 INFO crail: lookup: name /, success, fd 0
19/06/12 08:49:10 INFO crail: getDirectoryList: /
19/06/12 08:49:10 INFO crail: CoreInputStream: open, path /, fd 0, streamId 1, isDir true, readHint 0
19/06/12 08:49:10 INFO crail: Connecting to NVMf target at Transport address = /192.168.3.100:4420, subsystem NQN = nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/12 08:49:10 INFO disni: creating RdmaProvider of type 'nat'
19/06/12 08:49:10 INFO disni: jverbs jni version 32
19/06/12 08:49:10 INFO disni: sock_addr_in size mismatch, jverbs size 28, native size 16
19/06/12 08:49:10 INFO disni: IbvRecvWR size match, jverbs size 32, native size 32
19/06/12 08:49:10 INFO disni: IbvSendWR size mismatch, jverbs size 72, native size 128
19/06/12 08:49:10 INFO disni: IbvWC size match, jverbs size 48, native size 48
19/06/12 08:49:10 INFO disni: IbvSge size match, jverbs size 16, native size 16
19/06/12 08:49:10 INFO disni: Remote addr offset match, jverbs size 40, native size 40
19/06/12 08:49:10 INFO disni: Rkey offset match, jverbs size 48, native size 48
19/06/12 08:49:10 INFO disni: createEventChannel, objId 140229751834160
19/06/12 08:49:10 INFO disni: launching cm processor, cmChannel 0
19/06/12 08:49:10 INFO disni: createId, id 140229751892832
19/06/12 08:49:10 INFO disni: new client endpoint, id 0, idPriv 0
19/06/12 08:49:10 INFO disni: resolveAddr, addres /192.168.3.100:4420
19/06/12 08:49:10 INFO disni: got event type + UNKNOWN, srcAddress /0.0.0.0:0, dstAddress /192.168.3.100:4420
asqasq commented
Hi
Thanks for trying Crail. To narrow down the problem, let me ask some questions.
Is this an Infiniband or a RoCE network?
If I understand correctly, your Spark runs in a container with bridged
network and you see all RDMA devices.
Do you also run Crail in containers or do you run Crail natively on
physical hosts?
If you run Crail directly on physical hosts, does crail fs -ls / work, when
you execute it also on a physical host?
If you do a ib_send_bw test from the container you run Spark to the location,
where you run Crail, does it work? For example:
On the same node, as the Crail namenode runs (the "server"), do the following:
ib_send_bw -R
and in the Spark container, run:
ib_send_bw -R <IP of the "server" above>
Please let me know the outcome.
Thanks
Adrian
dacrespi commented
Hi Adrian, thanks for the quick response.
It appears to me that the code is looking for the interface to perhaps bind to (getting the IP address),
and the physical addresses are not in the containers (i.e. bridged network). If I knew what the code was
looking for, I could perhaps add that into the container. I was just starting to trace the code… but there
isn’t any debug output from this (i.e. not really an error), and there doesn’t appear to be any timeout
for it either. I left the program running for over 30 minutes just to be sure.
Answers are inline…
Regards,
David
From: Adrian Schuepbach<mailto:notifications@github.com>
Sent: Thursday, June 13, 2019 1:55 AM
To: zrlio/disni<mailto:disni@noreply.github.com>
Cc: David Crespi<mailto:david.crespi@storedgesystems.com>; Author<mailto:author@noreply.github.com>
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Hi
Thanks for trying Crail. To narrow down the problem, let me ask some questions.
Is this an Infiniband or a RoCE network?
DC: RoCE (Using Melanox rNIC’s)
If I understand correctly, your Spark runs in a container with bridged
network and you see all RDMA devices.
DC: yep
Do you also run Crail in containers or do you run Crail natively on
physical hosts?
DC: Crail Containers, but on a different server.
If you run Crail directly on physical hosts, does crail fs -ls / work, when
you execute it also on a physical host?
DC: running via containers, but that command does work fine. My test config used 1 Namenode, and 5 type 2 NVMf DataNodes.
If you do a ib_send_bw test from the container you run Spark to the location,
where you run Crail, does it work? For example:
On the same node, as the Crail namenode runs (the "server"), do the following:
ib_send_bw -R
and in the Spark container, run:
ib_send_bw -R <IP of the "server" above>
DC: I haven’t done the ib_send_bw test, but have been using the ping_pong test from inside each container, and these work find from inside the containers. I can try these tests today if that helps.
Server: ibv_rc_pingpong -s 4096 -d mlx5_1 -g 0
local address: LID 0x0000, QPN 0x0008c3, PSN 0xb16e32, GID fe80::9a03:9bff:fe98:9b23
remote address: LID 0x0000, QPN 0x0008db, PSN 0xa601e4, GID fe80::9a03:9bff:fe98:9b0f
8192000 bytes in 0.01 seconds = 10437.33 Mbit/sec
1000 iters in 0.01 seconds = 6.28 usec/iter
Client: ibv_rc_pingpong -d mlx5_1 -g 0 192.168.3.100
local address: LID 0x0000, QPN 0x0008db, PSN 0xa601e4, GID fe80::9a03:9bff:fe98:9b0f
remote address: LID 0x0000, QPN 0x0008c3, PSN 0xb16e32, GID fe80::9a03:9bff:fe98:9b23
8192000 bytes in 0.01 seconds = 12535.58 Mbit/sec
1000 iters in 0.01 seconds = 5.23 usec/iter
I’ve also tried to run these, and these never complete…
Client: java com.ibm.disni.examples.ReadClient -a 192.168.3.100 -p 1919
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/disni/target/disni-2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/jNVMf/target/jnvmf-1.5-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/crail/jars/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Server: java com.ibm.disni.examples.ReadServer -a 192.168.3.100
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/disni/target/disni-2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/jNVMf/target/jnvmf-1.5-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/crail/jars/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
ReadServer::server bound to address/192.168.3.100:1919
Please let me know the outcome.
Thanks
Adrian
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJI2YJINIGY2AI5XQ63P2IDPVA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXTAIMI#issuecomment-501613617>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJLHM24HS2BGVCCPBGTP2IDPVANCNFSM4HXKM4NQ>.
PepperJo commented
ibv_rc_pingpong does not use rdma connection management (which DiSNI is using) but manually changes QP state and uses GID/LID to connect. Can you please try to run rping in your container.
Regards,
Jonas
PepperJo commented
Check out this article on the MLX homepage: https://community.mellanox.com/s/article/howto-create-docker-container-enabled-with-roce
If you are using RoCE with CM you have to use host network:
Due to RDMA-CM limitations, the container must use the host network name space
dacrespi commented
Yes, I’m aware of that, but I was hoping there is way around that.
Spark wants to connect via hostnames, and when using the host network, well, its only the host,
so it fails to start up. Seems with all this technology these days, an rNIC should be handled just
like a NIC is. This RDMA stuff is really a problem. How do you guys run these configs? Are you running
on bare metal with your crail-spark?
Regards,
David
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Thursday, June 13, 2019 7:53:07 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Check out this article on the MLX homepage: https://community.mellanox.com/s/article/howto-create-docker-container-enabled-with-roce
If you are using RoCE with CM you have to use host network:
Due to RDMA-CM limitations, the container must use the host network name space
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJMTVK4DAO5FDGSEYA3P2JNNHA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXT6S7Q#issuecomment-501737854>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJNMX3XILF22A46RLNTP2JNNHANCNFSM4HXKM4NQ>.
dacrespi commented
Jonas,
I don’t know if you’ve seen this from Mellanox (pretty new):
https://hub.docker.com/r/rdma/sriov-plugin
This plugin brings the RoCE interface into the container without all the other stuff
that’s normally needed. The container still passes the ibv_rc_pingpong test, but also still
hangs in the crail code with the 0000 src address.
Regards,
David
From: Jonas Pfefferle<mailto:notifications@github.com>
Sent: Thursday, June 13, 2019 7:53 AM
To: zrlio/disni<mailto:disni@noreply.github.com>
Cc: David Crespi<mailto:david.crespi@storedgesystems.com>; Author<mailto:author@noreply.github.com>
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Check out this article on the MLX homepage: https://community.mellanox.com/s/article/howto-create-docker-container-enabled-with-roce
If you are using RoCE with CM you have to use host network:
Due to RDMA-CM limitations, the container must use the host network name space
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJMTVK4DAO5FDGSEYA3P2JNNHA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXT6S7Q#issuecomment-501737854>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJNMX3XILF22A46RLNTP2JNNHANCNFSM4HXKM4NQ>.
asqasq commented
Hi David
I tried this one out. As far as I remember, I read
that RoCE only works correctly with the "host network".
Indeed, ib_send_bw or ib_read_bw etc. did not work
correctly on a RoCE network.
I am working on containerizing Crail in a usable way
with RDMA support. So far, there are some limitations
(like the host network flag), which I also would
like to avoid.
So far, we run it mostly bare metal or the TCP version,
if we run it as containers.
Regards
Adrian
…On 14.06.2019 02:23, David Crespi wrote:
Jonas,
I don’t know if you’ve seen this from Mellanox (pretty new):
https://hub.docker.com/r/rdma/sriov-plugin
This plugin brings the RoCE interface into the container without all the
other stuff
that’s normally needed. The container still passes the ibv_rc_pingpong
test, but also still
hangs in the crail code with the 0000 src address.
Regards,
David
From: Jonas ***@***.***>
Sent: Thursday, June 13, 2019 7:53 AM
To: ***@***.***>
Cc: David ***@***.***>;
***@***.***>
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Check out this article on the MLX homepage:
https://community.mellanox.com/s/article/howto-create-docker-container-enabled-with-roce
If you are using RoCE with CM you have to use host network:
Due to RDMA-CM limitations, the container must use the host network name
space
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on
GitHub<#46?email_source=notifications&email_token=AK24SJMTVK4DAO5FDGSEYA3P2JNNHA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXT6S7Q#issuecomment-501737854>,
or mute the
thread<https://github.com/notifications/unsubscribe-auth/AK24SJNMX3XILF22A46RLNTP2JNNHANCNFSM4HXKM4NQ>.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#46?email_source=notifications&email_token=AHK75WGBT4OLIURRVZHRBA3P2LQI7A5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXVMEQI#issuecomment-501924417>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHK75WGLIJOZNK2QCNFDCTDP2LQI7ANCNFSM4HXKM4NQ>.
PepperJo commented
We know the pain with Spark and hostnames, there is a way to configure Spark with Yarn to use ips however it requires a Yarn configuration with a hard coded ip on every host (very cumbersome). I'm not an expert running Spark on container but it seems there is native support for containers in Spark, maybe Adrian can provide some more inside. Regarding the RNICs, we mostly run bare metal. When I ran with containers in the past I was using host network. This might also work: https://community.mellanox.com/s/article/docker-roce-macvlan-networking-with-connectx4-connectx5
I didn't know about the SRIOV plugin but again before trying to run Crail please give rping a try. It is part of the rdma CM examples.
Regards,
Jonas
dacrespi commented
I’ve tried macvlan as well as it seemed like an easier solution, but it hangs at the same point.
Seems like the plugin is the right solution (so far anyway) to this problem, it doesn’t require
the particular container to be running on the host network, the plugin is doing that for you.
Going through the code a bit now, there isn’t any logging or timeouts in this part of the
code (where the binding is attempting). Shouldn’t there at least be a timeout? Java is a
bit foreign to me however… so don’t take offence 😊
Regards,
David
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Friday, June 14, 2019 12:57:42 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
We know the pain with Spark and hostnames, there is a way to configure Spark with Yarn to use ips however it requires a Yarn configuration with a hard coded ip on every host (very cumbersome). I'm not an expert running Spark on container but it seems there is native support for containers in Spark, maybe Adrian can provide some more inside. Regarding the RNICs, we mostly run bare metal. When I ran with containers in the past I was using host network. This might also work: https://community.mellanox.com/s/article/docker-roce-macvlan-networking-with-connectx4-connectx5
Regards,
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJJXJHNBACPOAR6HTA3P2NFPNA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXWA6EA#issuecomment-502009616>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJO5EGDB4UEHHITOZADP2NFPNANCNFSM4HXKM4NQ>.
dacrespi commented
Guys,
I wanted to dig a bit through the code to see what was occurring and why it appeared to be hanging.
From what I can tell, the problem appears to be that a null memory address is being returned.
in the module ../verbs/impl/RdmaCmNat.java
At line 198, what’s being retuned is a -1 (null), which causes idPriv to be null, and then the while loop at
line 66 of RdmaCmProcessor.java will never stop, thus looking like a hang. I haven’t yet determined why
the null is returned, but I’m hoping it’s not related to the host network. I think this should be part of the
memory allocated in line 188?
Also, if the order of line 198 and 199 are switched, then the clientId get’s the null entry.
Thoughts?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
public RdmaCmEvent getCmEvent(RdmaEventChannel cmChannel, int timeout) throws IOException {
185 NatRdmaEventChannel channelImpl = (NatRdmaEventChannel) cmChannel;
186 RdmaCmEvent cmEvent = null;
187
188 MemBuf memBuf = memAlloc.allocate(2*8);
189
190 ByteBuffer buf = memBuf.getBuffer();
191
192 if (!channelImpl.isOpen()) {
193 throw new IOException("Trying to get CM event on closed channel.");
194 }
195 int event = nativeDispatcher._getCmEvent(channelImpl.getObjId(), memBuf.address(), memBuf.address() + 8, timeout);
196
197 if (event >= 0){
198 long _listenId = buf.getLong();
199 long _clientId = buf.getLong();
200 NatCmaIdPrivate idPriv = channelImpl.getCmId(_listenId);
201 NatCmaIdPrivate clientId = channelImpl.getCmId(_clientId);
202 if (event == RdmaCmEvent.EventType.RDMA_CM_EVENT_CONNECT_REQUEST.ordinal()){
203 clientId = new NatCmaIdPrivate(_clientId, channelImpl, nativeDispatcher);
204 clientId.setVerbs(idPriv.getVerbs());
205 channelImpl.addCmId(clientId);
206 }
207 cmEvent = new RdmaCmEvent(event, idPriv, clientId);
208 }
209
210 memBuf.free();
211
212 return cmEvent;
213 }
Regards,
David
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Friday, June 14, 2019 12:57:42 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
We know the pain with Spark and hostnames, there is a way to configure Spark with Yarn to use ips however it requires a Yarn configuration with a hard coded ip on every host (very cumbersome). I'm not an expert running Spark on container but it seems there is native support for containers in Spark, maybe Adrian can provide some more inside. Regarding the RNICs, we mostly run bare metal. When I ran with containers in the past I was using host network. This might also work: https://community.mellanox.com/s/article/docker-roce-macvlan-networking-with-connectx4-connectx5
Regards,
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJJXJHNBACPOAR6HTA3P2NFPNA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXWA6EA#issuecomment-502009616>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJO5EGDB4UEHHITOZADP2NFPNANCNFSM4HXKM4NQ>.
PepperJo commented
Hi David
Going through the code a bit now, there isn’t any logging or timeouts in this part of the
code (where the binding is attempting). Shouldn’t there at least be a timeout? Java is a
bit foreign to me however… so don’t take offence blush
I did not write this part of the code so I'm also not 100% sure if timeouts are missing or not. However, most of the RDMA CM functions do have a timeout argument, so I assume timeout is handled in the C code.
At line 198, what’s being retuned is a -1 (null), which causes idPriv to be null, and then the while loop at
line 66 of RdmaCmProcessor.java will never stop, thus looking like a hang. I haven’t yet determined why
the null is returned, but I’m hoping it’s not related to the host network. I think this should be part of the
memory allocated in line 188?
Looks like a bug to me.
David, I can only repeat myself: why not try to run some application that uses CM like rping before trying to run DiSNI. This way you can eliminate the possibility of bugs in DiSNI causing the hang.
Regards,
Jonas
patrickstuedi commented
Can we debug the simpler case of the RdmaReadServer/RdmaReadClient.
You mentioned you rand the read example. What is the error you get when running this?
server:
java -cp disni-1.7-jar-with-dependencies.jar:disni-1.7-tests.jar com.ibm.disni.examples.ReadServer -a 10.100.0.1
client:
java -cp disni-1.7-jar-with-dependencies.jar:disni-1.7-tests.jar com.ibm.disni.examples.ReadClient -a 10.100.0.1 -p 1919
Also, please try to run rping as Jonas's suggested and let us know the outcome.
Thanks
dacrespi commented
In the original post I send, I showed that the two containers were talking via
the ibv_rc_pingpong test. Does rping do more than that? I assumed it was the same thing.
Server
$ibv_rc_pingpong -g0 -d mlx5_1
local address: LID 0x0000, QPN 0x0008d6, PSN 0x97d7d3, GID fe80::9a03:9bff:fe98:9b23
remote address: LID 0x0000, QPN 0x0008dd, PSN 0x4ab8df, GID fe80::9a03:9bff:fe98:9b0f
8192000 bytes in 0.01 seconds = 11059.06 Mbit/sec
1000 iters in 0.01 seconds = 5.93 usec/iter
Client
$ibv_rc_pingpong -d mlx5_1 -g0 192.168.3.100 -s 4096
local address: LID 0x0000, QPN 0x0008dd, PSN 0x4ab8df, GID fe80::9a03:9bff:fe98:9b0f
remote address: LID 0x0000, QPN 0x0008d6, PSN 0x97d7d3, GID fe80::9a03:9bff:fe98:9b23
8192000 bytes in 0.01 seconds = 11808.29 Mbit/sec
1000 iters in 0.01 seconds = 5.55 usec/iter
I has also put in the original, that when I run the two tests jars, that it just sits there, just like
doing the crail command, but there isn’t any additional log info.
Server
$java -cp disni-2.0-jar-with-dependencies.jar:disni-2.0-tests.jar com.ibm.disni.examples.ReadServer -a 192.168.3.100
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
ReadServer::server bound to address/192.168.3.100:1919
Client
$ java -cp disni-2.0-jar-with-dependencies.jar:disni-2.0-tests.jar com.ibm.disni.examples.ReadClient -a 192.168.3.100 -p 1919
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Regards,
David
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Monday, June 17, 2019 12:39:26 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Hi David
Going through the code a bit now, there isn’t any logging or timeouts in this part of the
code (where the binding is attempting). Shouldn’t there at least be a timeout? Java is a
bit foreign to me however… so don’t take offence blush
I did not write this part of the code so I'm also not 100% sure if timeouts are missing or not. However, most of the RDMA CM functions do have a timeout argument, so I assume timeout is handled in the C code.
At line 198, what’s being retuned is a -1 (null), which causes idPriv to be null, and then the while loop at
line 66 of RdmaCmProcessor.java will never stop, thus looking like a hang. I haven’t yet determined why
the null is returned, but I’m hoping it’s not related to the host network. I think this should be part of the
memory allocated in line 188?
Looks like a bug to me.
David, I can only repeat myself: why not try to run some application that uses CM like rping before trying to run DiSNI. This way you can eliminate the possibility of bugs in DiSNI causing the hang.
Regards,
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJLTCFZ4VWZIXGZDPODP245S5A5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX2JY7Y#issuecomment-502570111>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJN7RSM34DLZUY77LRDP245S5ANCNFSM4HXKM4NQ>.
dacrespi commented
Well, it appears you’re correct, the rping does also fail, and in the address resolution (id_priv)
part of cma.c. So I guess I’ll dig into that for awhile. It would appear that, as you expected, it
it not your issue.
Regards,
David
From: David Crespi<mailto:david.crespi@storedgesystems.com>
Sent: Monday, June 17, 2019 8:38 AM
To: zrlio/disni<mailto:reply@reply.github.com>; zrlio/disni<mailto:disni@noreply.github.com>
Cc: Author<mailto:author@noreply.github.com>
Subject: RE: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
In the original post I send, I showed that the two containers were talking via
the ibv_rc_pingpong test. Does rping do more than that? I assumed it was the same thing.
Server
$ibv_rc_pingpong -g0 -d mlx5_1
local address: LID 0x0000, QPN 0x0008d6, PSN 0x97d7d3, GID fe80::9a03:9bff:fe98:9b23
remote address: LID 0x0000, QPN 0x0008dd, PSN 0x4ab8df, GID fe80::9a03:9bff:fe98:9b0f
8192000 bytes in 0.01 seconds = 11059.06 Mbit/sec
1000 iters in 0.01 seconds = 5.93 usec/iter
Client
$ibv_rc_pingpong -d mlx5_1 -g0 192.168.3.100 -s 4096
local address: LID 0x0000, QPN 0x0008dd, PSN 0x4ab8df, GID fe80::9a03:9bff:fe98:9b0f
remote address: LID 0x0000, QPN 0x0008d6, PSN 0x97d7d3, GID fe80::9a03:9bff:fe98:9b23
8192000 bytes in 0.01 seconds = 11808.29 Mbit/sec
1000 iters in 0.01 seconds = 5.55 usec/iter
I has also put in the original, that when I run the two tests jars, that it just sits there, just like
doing the crail command, but there isn’t any additional log info.
Server
$java -cp disni-2.0-jar-with-dependencies.jar:disni-2.0-tests.jar com.ibm.disni.examples.ReadServer -a 192.168.3.100
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
ReadServer::server bound to address/192.168.3.100:1919
Client
$ java -cp disni-2.0-jar-with-dependencies.jar:disni-2.0-tests.jar com.ibm.disni.examples.ReadClient -a 192.168.3.100 -p 1919
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Regards,
David
From: Jonas Pfefferle <notifications@github.com>
Sent: Monday, June 17, 2019 12:39:26 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Hi David
Going through the code a bit now, there isn’t any logging or timeouts in this part of the
code (where the binding is attempting). Shouldn’t there at least be a timeout? Java is a
bit foreign to me however… so don’t take offence blush
I did not write this part of the code so I'm also not 100% sure if timeouts are missing or not. However, most of the RDMA CM functions do have a timeout argument, so I assume timeout is handled in the C code.
At line 198, what’s being retuned is a -1 (null), which causes idPriv to be null, and then the while loop at
line 66 of RdmaCmProcessor.java will never stop, thus looking like a hang. I haven’t yet determined why
the null is returned, but I’m hoping it’s not related to the host network. I think this should be part of the
memory allocated in line 188?
Looks like a bug to me.
David, I can only repeat myself: why not try to run some application that uses CM like rping before trying to run DiSNI. This way you can eliminate the possibility of bugs in DiSNI causing the hang.
Regards,
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJLTCFZ4VWZIXGZDPODP245S5A5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX2JY7Y#issuecomment-502570111>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJN7RSM34DLZUY77LRDP245S5ANCNFSM4HXKM4NQ>.
patrickstuedi commented
Ok. So we narrowed it down to a raw RDMA/container problem. @asqasq Do you have further suggestions? I thought you had basic RDMA w CM working in containers..do you?
dacrespi commented
Well I thought so. Like I said, the Mellanox pingpong tests work find.
I am running crail type 2 on a differ server with the containers connected via the host network (as you suggested),
and that works fine. For flexibility, however, I would also like to run them NOT on the host network.
I’ve posted a bug report on the Mellanox site for their plugin. The plugin is supposed to export an IB interface
via SRIOV. And I do see all these interfaces in the containers now. I do think it’s odd that its failing in a very
similar place as with the crail code. It smells like there is a memory issue, not a IB interface issue.
I’m looking at the cma.c code now. It appears that they have had a few issues with race conditions because
of being in user space… maybe there is another?
Regards,
David
…________________________________
From: Patrick Stuedi <notifications@github.com>
Sent: Monday, June 17, 2019 12:01:41 PM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Ok. So we narrowed it down to a raw RDMA/container problem. @asqasq<https://github.com/asqasq> Do you have further suggestions? I thought you had basic RDMA w CM working in containers..do you?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJJEARUU4LV2GK2CKVTP27NRLA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX4EMCA#issuecomment-502810120>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJN3R5P5DOK27BZU32DP27NRLANCNFSM4HXKM4NQ>.
asqasq commented
Hi David
It's good to hear that, it is basically the same as I am seeing, this is why I suggested to run 'ib_send_bw -R' with CM initially.
At least when I tried it, Mellanox stated that it won't work with RoCE networks without the host flag )as I said earlier). We would also like to run it without host network, but apparently this is a known limitation (maybe a known bug).
At least we know now that it is not a DiSNi problem.
So far I dom't have a solution to that. Let's see what Mellanox replies.
Regards
Adrian
dacrespi commented
Thanks Adrian, you guys have been great with your support so far!
I’m looking forward to getting beyond this and getting things moving!
Regards,
David
…________________________________
From: Adrian Schuepbach <notifications@github.com>
Sent: Monday, June 17, 2019 12:45:20 PM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Hi David
It's good to hear that, it is basically the same as I am seeing, this is why I suggested to run 'ib_send_bw -R' with CM initially.
At least when I tried it, Mellanox stated that it won't work with RoCE networks without the host flag )as I said earlier). We would also like to run it without host network, but apparently this is a known limitation (maybe a known bug).
At least we know now that it is not a DiSNi problem.
So far I dom't have a solution to that. Let's see what Mellanox replies.
Regards
Adrian
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJJ4V52NOATXIB2V5O3P27SVBA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX4HY4A#issuecomment-502824048>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJIZE6DAXXVBCQUIDILP27SVBANCNFSM4HXKM4NQ>.
PepperJo commented
David,
Just to clarify, the difference between ibv_rc_pingpong and rping is that the former is manually connecting via GID/LID (using ibv_modify_qp) whereas the latter uses librdmacm to connect. Essentially the RDMA CM core tries to find the appropriate RDMA device by retrieving the MAC address which obviously in a bridged container case will not work. But I know namespace support was added to RDMA CM a while ago. According to this slidedeck from last year macvlan should work with CM (slide 13): http://qnib.org/data/isc2018/roce-containers.pdf
Jonas
dacrespi commented
Thanks for the link Jonas. I saw that presentation, which provided the motivation to get this
stuff to work. One of the presenters, Parav, has contacted me. After looking at the development work
in that code yesterday, I think I’m going to have to update to the latest kernel to get this to
work… going to confirm that today!
Regards,
David
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Tuesday, June 18, 2019 12:29:45 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
David,
Just to clarify, the difference between ibv_rc_pingpong and rping is that the former is manually connecting via GID/LID (using ibv_modify_qp) whereas the latter uses librdmacm to connect. Essentially the RDMA CM core tries to find the appropriate RDMA device by retrieving the MAC address which obviously in a bridged container case will not work. But I know namespace support was added to RDMA CM a while ago. According to this slidedeck from last year macvlan should work with CM (slide 13): http://qnib.org/data/isc2018/roce-containers.pdf<url>
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJKGTDHNBFW47JWTE5LP3CFGTA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX5OWDI#issuecomment-502983437>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJNKDFSF3U42VMOQFX3P3CFGTANCNFSM4HXKM4NQ>.
PepperJo commented
Great. Keep us updated if it works.
Jonas
dacrespi commented
Well, updating to at 5.0 kernel definitely helped, and the rping works now!
I got beyond that error, and a new one has shown up. But at least it does now connect.
19/06/18 07:07:06 INFO crail: lookup: name /, success, fd 0
19/06/18 07:07:06 INFO crail: getDirectoryList: /
19/06/18 07:07:06 INFO crail: CoreInputStream: open, path /, fd 0, streamId 1, isDir true, readHint 0
19/06/18 07:07:06 INFO crail: Connecting to NVMf target at Transport address = /192.168.3.100:4420, subsystem NQN = nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/18 07:07:06 INFO disni: creating RdmaProvider of type 'nat'
19/06/18 07:07:06 INFO disni: jverbs jni version 32
19/06/18 07:07:06 INFO disni: sock_addr_in size mismatch, jverbs size 28, native size 16
19/06/18 07:07:06 INFO disni: IbvRecvWR size match, jverbs size 32, native size 32
19/06/18 07:07:06 INFO disni: IbvSendWR size mismatch, jverbs size 72, native size 128
19/06/18 07:07:06 INFO disni: IbvWC size match, jverbs size 48, native size 48
19/06/18 07:07:06 INFO disni: IbvSge size match, jverbs size 16, native size 16
19/06/18 07:07:06 INFO disni: Remote addr offset match, jverbs size 40, native size 40
19/06/18 07:07:06 INFO disni: Rkey offset match, jverbs size 48, native size 48
19/06/18 07:07:06 INFO disni: createEventChannel, objId 139815421662496
19/06/18 07:07:06 INFO disni: launching cm processor, cmChannel 0
19/06/18 07:07:06 INFO disni: createId, id 139815421750560
19/06/18 07:07:06 INFO disni: new client endpoint, id 0, idPriv 0
19/06/18 07:07:06 INFO disni: resolveAddr, addres /192.168.3.100:4420
19/06/18 07:07:06 INFO disni: resolveRoute, id 0
19/06/18 07:07:06 INFO disni: allocPd, objId 139815421769008
19/06/18 07:07:06 INFO disni: setting up protection domain, context 106, pd 1
19/06/18 07:07:06 INFO disni: new endpoint CQ processor
19/06/18 07:07:06 INFO disni: createCompChannel, context 139809372819024
19/06/18 07:07:06 INFO disni: createCQ, objId 139815421765824, ncqe 64
19/06/18 07:07:06 INFO disni: createQP, objId 139815421763688, send_wr size 32, recv_wr_size 32
19/06/18 07:07:06 INFO disni: connect, id 0
19/06/18 07:07:06 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:38768, dstAddress /192.168.3.100:4420
19/06/18 07:07:06 INFO crail: Cannot close, pending operations, opcount 1, path /
19/06/18 07:07:06 INFO crail: error when closing directory stream java.io.IOException: Cannot close, pending operations, opcount 1
19/06/18 07:07:06 INFO crail: Closing CrailHadoopFileSystem
19/06/18 07:07:06 INFO crail: Closing CrailFS singleton
19/06/18 07:07:06 INFO crail: Cannot close, pending operations, opcount 1, path /
java.io.IOException: java.io.IOException: Cannot close, pending operations, opcount 1
at org.apache.crail.CrailStore.close(CrailStore.java:55)
at org.apache.crail.hdfs.CrailHadoopFileSystem.close(CrailHadoopFileSystem.java:290)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)
at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
Caused by: java.io.IOException: Cannot close, pending operations, opcount 1
at org.apache.crail.core.CoreInputStream.close(CoreInputStream.java:108)
at org.apache.crail.core.CoreDataStore.closeFileSystem(CoreDataStore.java:515)
at org.apache.crail.CrailStore.close(CrailStore.java:52)
... 4 more
19/06/18 07:07:06 INFO FileSystem: FileSystem.Cache.closeAll() threw an exception:
java.io.IOException: java.io.IOException: java.io.IOException: Cannot close, pending operations, opcount 1
Regards,
David
C: 714-476-2692
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Tuesday, June 18, 2019 6:32:12 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Great. Keep us updated if it works.
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJKVDAGQJUGSYTWI6XDP3DPVZA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX6UEKA#issuecomment-503136808>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJLR7PTAJPWYHDNTFQ3P3DPVZANCNFSM4HXKM4NQ>.
PepperJo commented
Good to hear! Looks like it could connect to the namenode but not to the datanode (File names are stored in directory files on datanodes). Can you make sure that the datanode is accessible from within the container?
Jonas
dacrespi commented
Jonas,
I didn’t set the slicesize. I just set it to 65536 and everything worked!!!! YEAH!!!
Regards,
David
C: 714-476-2692
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Tuesday, June 18, 2019 7:26:53 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Looks like it could connect to the namenode but not to the datanode (File names are stored in directory files on datanodes). Can you make sure that the datanode is accessible from within the container?
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJJ4VZ6MWFEC74QTGZ3P3DWC3A5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX6ZZ4Y#issuecomment-503160051>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJOJ7OU2EVKGU6WFVCLP3DWC3ANCNFSM4HXKM4NQ>.
PepperJo commented
Nice! I just noticed that you use the NVMf storage tier. Be aware that blocksize and slicesize have to be multiple of sector size of your SSD. Also directory record entry size has to be sector size otherwise we cannot guarantee atomicity (default is 512 IIRC).
Let us know if you have any more questions.
Jonas
dacrespi commented
You guys have been super helpful, thanks for all the support.
Now on to the shuffle 😊
I did have a question before you go… The documentation states that…
“Since Spark version 2.0.0, broadcast is no longer an exchangeable plugin, unfortunately. To use the Crail broadcast plugin in Spark it has to be manually added to Spark’s BroadcastManager.scala.”
So we need to change the scala code within the spark code to achieve this? If yes, would you happen to have sample
Code to accomplish this?
Regards,
David
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Tuesday, June 18, 2019 7:31:22 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Nice! I just noticed that you use the NVMf storage tier. Be aware that blocksize and slicesize have to be multiple of sector size of your SSD. Also directory record entry size have to be sector size otherwise we cannot guarantee atomicity (default is 512 IIRC).
Let us know if you have any more questions.
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJNECQMWQSMPGHUHIWTP3DWTVA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX62JXI#issuecomment-503162077>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJL7FW2IEM7RCCPOUALP3DWTVANCNFSM4HXKM4NQ>.
PepperJo commented
The shuffle plugin is independent of broadcast. There are a few applications that use broadcast extensively, e.g. like SQL otherwise you probably will not see a big difference.
Jonas
dacrespi commented
Awesome. You guys planning on presenting in the states anytime soon?
Regards,
David
C: 714-476-2692
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Tuesday, June 18, 2019 7:37:47 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
The shuffle plugin is independent of broadcast. There are a few applications that use broadcast extensively, e.g. like SQL otherwise you probably will not see a big difference.
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJMPU3DDS2MFKX5I6BLP3DXLXA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX63AEY#issuecomment-503164947>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJMIAZHR5Q3GOHGLR7TP3DXLXANCNFSM4HXKM4NQ>.
PepperJo commented
Yes, Patrick will be at Usenix ATC: https://www.usenix.org/conference/atc19/presentation/stuedi
dacrespi commented
Great, and thanks again for the assist!!!
Regards,
David
C: 714-476-2692
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Tuesday, June 18, 2019 7:40:25 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
Yes, Patrick will be at Usenix ATC: https://www.usenix.org/conference/atc19/presentation/stuedi
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJOBYZEW3B5XBYOTPWLP3DXVTA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX63J5Y#issuecomment-503166199>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJKJX4DG7LJN5CINEZTP3DXVTANCNFSM4HXKM4NQ>.
PepperJo commented
No problem. I will close this issue, feel free to open a new one or open a JIRA ticket here: https://issues.apache.org/jira/projects/CRAIL/issues if the problem is Crail related.
Regards,
Jonas
dacrespi commented
Jonas,
Another question…
I’m running terasort. It passes using crail as the storage, but
It doesn’t appear to be calling the close after each part of terasort.
When I do a cntl-c, the close is called and then it moves on to the
next operation. Is that correct behavior? When I use HDFS, it calls
the close.
Regards,
David
C: 714-476-2692
…________________________________
From: Jonas Pfefferle <notifications@github.com>
Sent: Tuesday, June 18, 2019 7:43:08 AM
To: zrlio/disni
Cc: David Crespi; Author
Subject: Re: [zrlio/disni] UNKNOWN, srcAddress /0.0.0.0:0 (#46)
No problem. I will close this issue, feel free to open a new one or open a JIRA ticket here: https://issues.apache.org/jira/projects/CRAIL/issues<url> if the problem is Crail related.
Regards,
Jonas
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AK24SJM72FSTBZRVGKDDTYLP3DX7ZA5CNFSM4HXKM4N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX63UDY#issuecomment-503167503>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK24SJORQ4RUUX4BUTGLL5TP3DX7ZANCNFSM4HXKM4NQ>.
patrickstuedi commented
Hi David,
I do remember HDFS adaptor on Crail not being closed properly in Spark runs using Crail as input or output. We should have looked into this long time ago I guess. I think the problem is that Spark can deal with multiple file system objects and keeps them cached and somehow we appear to not catch the close trigger properly.
Would you mind re-posting this on the crail mailing list which would be the right place to discuss those things (see http://crail.incubator.apache.org/community for the mailing list, dev list is the right one). Maybe someone can help there.
Cheers,
Patrick