-Re: Hadoop on IPv6
Adam Muise 2013-09-10, 16:34
Harsh is giving you a best practice for JVMs using IPv4 in general. As what
I am suggesting is IPv4-only connections to the Hadoop daemons and clients
on the cluster and gateway, you would not have issues heeding his advice.
Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
client connecting to the proxy server as the proxy would expect IPv6 and
you stated that's what you wanted your network to run on. The assumption is
that any application or client outside of the cluster and gateway node only
I suggest you identify how your clients access the Hadoop cluster (based on
Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
gateway node or simply log into cluster boxes, and what applications rely
on connections to this cluster. That is your list of hosts and ports that
you will have to find a network solution for or hide behind a gateway node.
Make your decisions around this.
Remember that my proxy suggestion only worked for RESTful web based traffic
and that's not the entire stack that typically gets used in Hadoop.
ODBC/JDBC does NOT fall under this category for example. If you are
upgrading your network already, ask the network specialists or consultants
about NAT64 and equivalent options.
On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
> We have the config params that you mentioned, but to be clear are you
> saying that the suggestions Adam made won't work because of these
> configurations ?
> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> For Hadoop JVMs, generally speaking, need to have the following in
>> your hadoop-env.sh for HADOOP_OPTS:
>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>> We also ship with this pre-added in the default configs of more recent
>> You may also need these on other JVM based daemons' opts, such as
>> HBASE_OPTS/etc. depending on what all you run.
>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <[EMAIL PROTECTED]> wrote:
>> > Adam,
>> > Thank you for the suggestion. Could you point me to any documentation
>> > can help with the configuration changes I need to make? Thanks.
>> > Aji
>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <[EMAIL PROTECTED]>
>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> >> isolated from the rest of your network for security and performance
>> >> already anyway.
>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> >> even setup a web proxy for some of the required sites using nginx or
>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> >> it is fully supported and well-tested.
>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>> >> --
>> >> Adam Muise
>> >> Solution Engineer
>> >> Hortonworks
>> >> [EMAIL PROTECTED]
>> >> 416-417-4037
>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>> >> Hortonworks Virtual Sandbox
>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> >> to which it is addressed and may contain information that is
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >> this message is not the intended recipient, you are hereby notified
>> that any
>> >> printing, copying, dissemination, distribution, disclosure or
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.