Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Re: SDP support for Hadoop RPC


+
Milind Bhandarkar 2013-10-09, 00:57
+
Yasin Celik 2013-10-09, 01:13
+
Milind Bhandarkar 2013-10-09, 01:19
+
Steve Loughran 2013-10-09, 08:45
+
Milind Bhandarkar 2013-10-09, 09:09
+
Suresh Srinivas 2013-10-09, 15:47
+
Milind Bhandarkar 2013-10-09, 17:38
+
Steve Loughran 2013-10-09, 08:50
+
Luke Lu 2013-10-10, 08:00
+
Steve Loughran 2013-10-10, 08:41
Copy link to this message
-
Re: SDP support for Hadoop RPC
Hi Luke,

I actually tested hadoop RPC using SDP. But it did not work for me.
I used hadoop-core 0.20.2
Following is the error I received:

java.io.IOException: Address family not supported by protocol
    at sun.net.sdp.SdpSupport.
convert0(Native Method)
    at sun.net.sdp.SdpSupport.convertSocket(SdpSupport.java:70)
    at sun.net.sdp.SdpProvider.convertTcpToSdpIfMatch(SdpProvider.java:300)
    at sun.net.sdp.SdpProvider.implBeforeTcpBind(SdpProvider.java:323)
    at sun.net.NetHooks.beforeTcpBind(NetHooks.java:86)
    at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:198)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.apache.hadoop.ipc.Server.bind(Server.java:188)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)

My sdp.conf file is like:
bind * *
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050

I also tried this for one specific node:

bind 10.80.70.13 50031
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
Thank you very much.
On Thu, Oct 10, 2013 at 4:00 AM, Luke Lu <[EMAIL PROTECTED]> wrote:

> Hadoop works with Java 7, which has builtin SDP support. You'll need to
> pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
> JVMs via env var HADOOP_OPTS.
>
> One caveat of SDP is the 64KB message size (performance cliff beyond 64KB),
> which _should_ work with default hadoop config, where dfs packet size is
> 64KB.
>
> __Luke
>
>
> On Tue, Oct 8, 2013 at 4:49 PM, Yasin Celik <[EMAIL PROTECTED]>
> wrote:
>
> > Hello,
> >
> > I looked around but I think it seems there is no Sockets Direct
> > Protocol(SDP) support in Hadoop RPC.
> > I basically want to take advantage of Infiniband when transferring data.
> > Is there any known support for this? Has anybody tried this before?
> >
> > Best
> >
> > ---------
> > Yasin Celik
> >
>

--
---------
Yasin Celik
+
Luke Lu 2013-10-10, 19:12