Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Need help regarding HDFS-RAID


+
Ajit Ratnaparkhi 2011-09-15, 11:07
+
Harsh J 2011-09-15, 11:35
+
Ajit Ratnaparkhi 2011-09-15, 12:31
+
Dhruba Borthakur 2011-09-15, 17:06
Copy link to this message
-
Re: Need help regarding HDFS-RAID
Andrew Purtell 2011-09-15, 17:08
But that is the HDFS RAID effectively in 0.22+, not 0.21, right Dhruba?

 
Best regards,
       - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Dhruba Borthakur <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Thursday, September 15, 2011 10:06 AM
>Subject: Re: Need help regarding HDFS-RAID
>
>
>We use HDFS RAID in a big way. Data older than 12 days are RAIDED using XOR encoding (effective replication of 2.5). Data older than a few months are raided using ReedSolomon (effective observed replication factor of 1.5). This is running on our 60 PB size cluster for about an year now.
>
>
>thanks
>dhruba
>
>
>
>On Thu, Sep 15, 2011 at 5:31 AM, Ajit Ratnaparkhi <[EMAIL PROTECTED]> wrote:
>
>Hi,
>>
>>
>>We were planning to use it for past data archival(instead of moving it to archival store).
>>Archiving it in HDFS gives advantage of making it easily available for processing whenever required.
>>
>>
>>Is there any archival solution in hadoop ecosystem?
>>
>>
>>thanks,
>>Ajit.
>>
>>
>>
>>On Thu, Sep 15, 2011 at 5:05 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>Hey Ajit,
>>>
>>>HDFS-RAID was never part of the 0.20 release. It made its debut in the
>>>0.21 release [1]. I know that Facebook uses it (and also did develop
>>>it), but unsure of users beyond Facebook.
>>>
>>>While 0.21 overall is not entirely deemed as production-usable yet
>>>(and is in fact, possibly abandoned for efforts on 0.22+), you can
>>>give that release a whirl on a test cluster and see for yourself if
>>>your need beats the stability.
>>>
>>>Just curious though - why are you looking to use this specifically?
>>>
>>>[1] - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21/mapreduce/src/contrib/raid/
>>>
>>>
>>>On Thu, Sep 15, 2011 at 4:37 PM, Ajit Ratnaparkhi
>>><[EMAIL PROTECTED]> wrote:
>>>> Hi,
>>>> We want to use HDFS-RAID in our production cluster.
>>>> (http://wiki.apache.org/hadoop/HDFS-RAID)
>>>> I am not able to find source/binaries/configs for this in official hadoop
>>>> distribution from apache hadoop. (checked in 0.20.1 and 0.20.2).
>>>> Can somebody please tell me where can I find that? and installation
>>>> procedure?
>>>> Also, is HDFS-RAID implementation stable enough to use in production?
>>>> thanks,
>>>> Ajit.
>>>>
>>>
>>>
>>>
>>>--
>>>Harsh J
>>>
>>
>
>
>
>--
>Connect to me at http://www.facebook.com/dhruba
>
>
>
+
Dhruba Borthakur 2011-09-15, 17:14
+
Ajit Ratnaparkhi 2011-09-15, 17:54
+
Andrew Purtell 2011-09-15, 18:01
+
Ajit Ratnaparkhi 2011-09-16, 05:43
+
Andrew Purtell 2011-09-17, 16:16
+
Dhruba Borthakur 2011-09-20, 09:18
+
Ajit Ratnaparkhi 2011-09-20, 13:49
+
Andrew Purtell 2011-09-20, 16:03
+
Dhruba Borthakur 2011-09-20, 16:49
+
Andrew Purtell 2011-09-20, 23:10