Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Hadoop hardware failure recovery


+
Aji Janis 2012-08-10, 18:38
+
Ted Dunning 2012-08-10, 18:55
+
anil gupta 2012-08-10, 19:12
+
Mohammad Tariq 2012-08-10, 19:16
+
Harsh J 2012-08-12, 12:43
+
Arun C Murthy 2012-08-12, 19:07
Copy link to this message
-
Re: Hadoop hardware failure recovery
Aji Janis 2012-08-13, 13:57
Thank you everyone for all the feedback and suggestions. Its good to know
these details as I move forward.

Piling on to the question, I am curious if any of you have experience with
Accumulo (a requirement for me hence not optional). I was wondering if the
data loss (physical crash of the hard drive) in this case would be resolved
by Hadoop (HDFS I should say). Any suggestions and/or where I could find
some specs on this would be really appreciated!
Thank you again for all the pointers.
-Aji

On Sun, Aug 12, 2012 at 3:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> Yep, hadoop-2 is alpha but is progressing nicely...
>
> However, if you have access to some 'enterprise HA' utilities (VMWare or
> Linux HA) you can get *very decent* production-grade high-availability in
> hadoop-1.x too (both NameNode for HDFS and JobTracker for MapReduce).
>
> Arun
>
> On Aug 10, 2012, at 12:12 PM, anil gupta wrote:
>
> Hi Aji,
>
> Adding onto whatever Mohammad Tariq said, If you use Hadoop 2.0.0-Alpha
> then Namenode is not a single point of failure.However, Hadoop 2.0.0 is not
> of production quality yet(its in Alpha).
> Namenode use to be a Single Point of Failure in releases prior to Hadoop
> 2.0.0.
>
> HTH,
> Anil Gupta
>
> On Fri, Aug 10, 2012 at 11:55 AM, Ted Dunning <[EMAIL PROTECTED]>wrote:
>
>> Hadoop's file system was (mostly) copied from the concepts of Google's
>> old file system.
>>
>> The original paper is probably the best way to learn about that.
>>
>> http://research.google.com/archive/gfs.html
>>
>>
>>
>> On Fri, Aug 10, 2012 at 11:38 AM, Aji Janis <[EMAIL PROTECTED]> wrote:
>>
>>> I am very new to Hadoop. I am considering setting up a Hadoop cluster
>>> consisting of 5 nodes where each node has 3 internal hard drives. I
>>> understand HDFS has a configurable redundancy feature but what happens if
>>> an entire drive crashes (physically) for whatever reason? How does Hadoop
>>> recover, if it can, from this situation? What else should I know before
>>> setting up my cluster this way? Thanks in advance.
>>>
>>>
>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
+
Harsh J 2012-08-13, 14:55
+
Mohammad Tariq 2012-08-10, 18:43
+
Harsh J 2012-08-12, 12:45
+
Mohammad Tariq 2012-08-12, 17:47
+
Harsh J 2012-08-13, 15:42
+
Steve Loughran 2012-08-13, 16:10
+
Jeffrey Buell 2012-08-14, 00:15