Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Hadoop on IPv6


+
Aji Janis 2013-09-09, 17:19
+
Adam Muise 2013-09-09, 17:30
+
Aji Janis 2013-09-09, 20:35
+
Harsh J 2013-09-09, 20:55
+
Adam Muise 2013-09-09, 20:48
+
Aji Janis 2013-09-09, 20:50
+
Adam Muise 2013-09-10, 16:34
Copy link to this message
-
Re: Hadoop on IPv6
Thank you for the clarification Adam.
On Tue, Sep 10, 2013 at 12:34 PM, Adam Muise <[EMAIL PROTECTED]> wrote:

> Harsh is giving you a best practice for JVMs using IPv4 in general. As
> what I am suggesting is IPv4-only connections to the Hadoop daemons and
> clients on the cluster and gateway, you would not have issues heeding his
> advice.
>
> Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
> client connecting to the proxy server as the proxy would expect IPv6 and
> you stated that's what you wanted your network to run on. The assumption is
> that any application or client outside of the cluster and gateway node only
> uses IPv6.
>
> I suggest you identify how your clients access the Hadoop cluster (based
> on Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
> gateway node or simply log into cluster boxes, and what applications rely
> on connections to this cluster. That is your list of hosts and ports that
> you will have to find a network solution for or hide behind a gateway node.
> Make your decisions around this.
>
> Remember that my proxy suggestion only worked for RESTful web based
> traffic and that's not the entire stack that typically gets used in Hadoop.
> ODBC/JDBC does NOT fall under this category for example. If you are
> upgrading your network already, ask the network specialists or consultants
> about NAT64 and equivalent options.
>
>
> On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
>
>> Harsh,
>>
>>  We have the config params that you mentioned, but to be clear are you
>> saying that the suggestions Adam made won't work because of these
>> configurations ?
>>
>> Thanks,
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> For Hadoop JVMs, generally speaking, need to have the following in
>>> your hadoop-env.sh for HADOOP_OPTS:
>>>
>>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>>
>>> We also ship with this pre-added in the default configs of more recent
>>> releases.
>>>
>>> You may also need these on other JVM based daemons' opts, such as
>>> HBASE_OPTS/etc. depending on what all you run.
>>>
>>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <[EMAIL PROTECTED]> wrote:
>>> > Adam,
>>> > Thank you for the suggestion. Could you point me to any documentation
>>> that
>>> > can help with the configuration changes I need to make? Thanks.
>>> >
>>> > Aji
>>> >
>>> >
>>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <[EMAIL PROTECTED]>
>>> wrote:
>>> >>
>>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should
>>> be
>>> >> isolated from the rest of your network for security and performance
>>> reasons
>>> >> already anyway.
>>> >>
>>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>>> via
>>> >> the ipv6 network and use the cluster via ipv4 on the gateway node.
>>> You could
>>> >> even setup a web proxy for some of the required sites using nginx or
>>> your
>>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop
>>> cluster until
>>> >> it is fully supported and well-tested.
>>> >>
>>> >>
>>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
>>> >>>
>>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>>> Currently,
>>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >>
>>> >> Adam Muise
>>> >> Solution Engineer
>>> >> Hortonworks
>>> >> [EMAIL PROTECTED]
>>> >> 416-417-4037
>>> >>
>>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop.
>>> >>
>>> >> Hortonworks Virtual Sandbox
>>> >>
>>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>>> >>
>>> >> CONFIDENTIALITY NOTICE
>>> >> NOTICE: This message is intended for the use of the individual or
>>> entity
>>> >> to which it is addressed and may contain information that is