Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Hadoop on IPv6


+
Aji Janis 2013-09-09, 17:19
+
Adam Muise 2013-09-09, 17:30
+
Aji Janis 2013-09-09, 20:35
+
Harsh J 2013-09-09, 20:55
+
Adam Muise 2013-09-09, 20:48
+
Aji Janis 2013-09-09, 20:50
+
Adam Muise 2013-09-10, 16:34
Copy link to this message
-
Re: Hadoop on IPv6
Thank you for the clarification Adam.
On Tue, Sep 10, 2013 at 12:34 PM, Adam Muise <[EMAIL PROTECTED]> wrote:

> Harsh is giving you a best practice for JVMs using IPv4 in general. As
> what I am suggesting is IPv4-only connections to the Hadoop daemons and
> clients on the cluster and gateway, you would not have issues heeding his
> advice.
>
> Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
> client connecting to the proxy server as the proxy would expect IPv6 and
> you stated that's what you wanted your network to run on. The assumption is
> that any application or client outside of the cluster and gateway node only
> uses IPv6.
>
> I suggest you identify how your clients access the Hadoop cluster (based
> on Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
> gateway node or simply log into cluster boxes, and what applications rely
> on connections to this cluster. That is your list of hosts and ports that
> you will have to find a network solution for or hide behind a gateway node.
> Make your decisions around this.
>
> Remember that my proxy suggestion only worked for RESTful web based
> traffic and that's not the entire stack that typically gets used in Hadoop.
> ODBC/JDBC does NOT fall under this category for example. If you are
> upgrading your network already, ask the network specialists or consultants
> about NAT64 and equivalent options.
>
>
> On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
>
>> Harsh,
>>
>>  We have the config params that you mentioned, but to be clear are you
>> saying that the suggestions Adam made won't work because of these
>> configurations ?
>>
>> Thanks,
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> For Hadoop JVMs, generally speaking, need to have the following in
>>> your hadoop-env.sh for HADOOP_OPTS:
>>>
>>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>>
>>> We also ship with this pre-added in the default configs of more recent
>>> releases.
>>>
>>> You may also need these on other JVM based daemons' opts, such as
>>> HBASE_OPTS/etc. depending on what all you run.
>>>
>>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <[EMAIL PROTECTED]> wrote:
>>> > Adam,
>>> > Thank you for the suggestion. Could you point me to any documentation
>>> that
>>> > can help with the configuration changes I need to make? Thanks.
>>> >
>>> > Aji
>>> >
>>> >
>>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <[EMAIL PROTECTED]>
>>> wrote:
>>> >>
>>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should
>>> be
>>> >> isolated from the rest of your network for security and performance
>>> reasons
>>> >> already anyway.
>>> >>
>>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>>> via
>>> >> the ipv6 network and use the cluster via ipv4 on the gateway node.
>>> You could
>>> >> even setup a web proxy for some of the required sites using nginx or
>>> your
>>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop
>>> cluster until
>>> >> it is fully supported and well-tested.
>>> >>
>>> >>
>>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <[EMAIL PROTECTED]> wrote:
>>> >>>
>>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>>> Currently,
>>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >>
>>> >> Adam Muise
>>> >> Solution Engineer
>>> >> Hortonworks
>>> >> [EMAIL PROTECTED]
>>> >> 416-417-4037
>>> >>
>>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop.
>>> >>
>>> >> Hortonworks Virtual Sandbox
>>> >>
>>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>>> >>
>>> >> CONFIDENTIALITY NOTICE
>>> >> NOTICE: This message is intended for the use of the individual or
>>> entity
>>> >> to which it is addressed and may contain information that is
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB