Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Hadoop hardware failure recovery


+
Aji Janis 2012-08-10, 18:38
+
Ted Dunning 2012-08-10, 18:55
+
anil gupta 2012-08-10, 19:12
+
Mohammad Tariq 2012-08-10, 19:16
+
Harsh J 2012-08-12, 12:43
+
Arun C Murthy 2012-08-12, 19:07
Copy link to this message
-
Re: Hadoop hardware failure recovery
Thank you everyone for all the feedback and suggestions. Its good to know
these details as I move forward.

Piling on to the question, I am curious if any of you have experience with
Accumulo (a requirement for me hence not optional). I was wondering if the
data loss (physical crash of the hard drive) in this case would be resolved
by Hadoop (HDFS I should say). Any suggestions and/or where I could find
some specs on this would be really appreciated!
Thank you again for all the pointers.
-Aji

On Sun, Aug 12, 2012 at 3:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> Yep, hadoop-2 is alpha but is progressing nicely...
>
> However, if you have access to some 'enterprise HA' utilities (VMWare or
> Linux HA) you can get *very decent* production-grade high-availability in
> hadoop-1.x too (both NameNode for HDFS and JobTracker for MapReduce).
>
> Arun
>
> On Aug 10, 2012, at 12:12 PM, anil gupta wrote:
>
> Hi Aji,
>
> Adding onto whatever Mohammad Tariq said, If you use Hadoop 2.0.0-Alpha
> then Namenode is not a single point of failure.However, Hadoop 2.0.0 is not
> of production quality yet(its in Alpha).
> Namenode use to be a Single Point of Failure in releases prior to Hadoop
> 2.0.0.
>
> HTH,
> Anil Gupta
>
> On Fri, Aug 10, 2012 at 11:55 AM, Ted Dunning <[EMAIL PROTECTED]>wrote:
>
>> Hadoop's file system was (mostly) copied from the concepts of Google's
>> old file system.
>>
>> The original paper is probably the best way to learn about that.
>>
>> http://research.google.com/archive/gfs.html
>>
>>
>>
>> On Fri, Aug 10, 2012 at 11:38 AM, Aji Janis <[EMAIL PROTECTED]> wrote:
>>
>>> I am very new to Hadoop. I am considering setting up a Hadoop cluster
>>> consisting of 5 nodes where each node has 3 internal hard drives. I
>>> understand HDFS has a configurable redundancy feature but what happens if
>>> an entire drive crashes (physically) for whatever reason? How does Hadoop
>>> recover, if it can, from this situation? What else should I know before
>>> setting up my cluster this way? Thanks in advance.
>>>
>>>
>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
+
Harsh J 2012-08-13, 14:55
+
Mohammad Tariq 2012-08-10, 18:43
+
Harsh J 2012-08-12, 12:45
+
Mohammad Tariq 2012-08-12, 17:47
+
Harsh J 2012-08-13, 15:42
+
Steve Loughran 2012-08-13, 16:10
+
Jeffrey Buell 2012-08-14, 00:15
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB