Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Re: SDP support for Hadoop RPC


Copy link to this message
-
Re: SDP support for Hadoop RPC
Hi Luke,

I actually tested hadoop RPC using SDP. But it did not work for me.
I used hadoop-core 0.20.2
Following is the error I received:

java.io.IOException: Address family not supported by protocol
    at sun.net.sdp.SdpSupport.
convert0(Native Method)
    at sun.net.sdp.SdpSupport.convertSocket(SdpSupport.java:70)
    at sun.net.sdp.SdpProvider.convertTcpToSdpIfMatch(SdpProvider.java:300)
    at sun.net.sdp.SdpProvider.implBeforeTcpBind(SdpProvider.java:323)
    at sun.net.NetHooks.beforeTcpBind(NetHooks.java:86)
    at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:198)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.apache.hadoop.ipc.Server.bind(Server.java:188)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)

My sdp.conf file is like:
bind * *
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050

I also tried this for one specific node:

bind 10.80.70.13 50031
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
Thank you very much.
On Thu, Oct 10, 2013 at 4:00 AM, Luke Lu <[EMAIL PROTECTED]> wrote:

> Hadoop works with Java 7, which has builtin SDP support. You'll need to
> pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
> JVMs via env var HADOOP_OPTS.
>
> One caveat of SDP is the 64KB message size (performance cliff beyond 64KB),
> which _should_ work with default hadoop config, where dfs packet size is
> 64KB.
>
> __Luke
>
>
> On Tue, Oct 8, 2013 at 4:49 PM, Yasin Celik <[EMAIL PROTECTED]>
> wrote:
>
> > Hello,
> >
> > I looked around but I think it seems there is no Sockets Direct
> > Protocol(SDP) support in Hadoop RPC.
> > I basically want to take advantage of Infiniband when transferring data.
> > Is there any known support for this? Has anybody tried this before?
> >
> > Best
> >
> > ---------
> > Yasin Celik
> >
>

--
---------
Yasin Celik
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB