Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Need help regarding HDFS-RAID


+
Ajit Ratnaparkhi 2011-09-15, 11:07
+
Harsh J 2011-09-15, 11:35
+
Ajit Ratnaparkhi 2011-09-15, 12:31
+
Dhruba Borthakur 2011-09-15, 17:06
+
Andrew Purtell 2011-09-15, 17:08
+
Dhruba Borthakur 2011-09-15, 17:14
+
Ajit Ratnaparkhi 2011-09-15, 17:54
Copy link to this message
-
Re: Need help regarding HDFS-RAID
HDFS RAID from 0.21 will work if back ported to 0.20. Only a minor fixup is needed.

HDFS RAID from 0.22 relies on new HDFS APIs not available in 0.20.

 
Best regards,
    - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Ajit Ratnaparkhi <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Cc: Andrew Purtell <[EMAIL PROTECTED]>
>Sent: Thursday, September 15, 2011 10:54 AM
>Subject: Re: Need help regarding HDFS-RAID
>
>
>Thanks for the info!
>So can I use HDFS-RAID taken from apache hdfs trunk as it is with hadoop-0.20.1/hadoop-0.20.2 ? It seems to be under branch 0.21, will it work with 0.20.* ?
>
>
>thanks,
>-Ajit.
>
>
>On Thu, Sep 15, 2011 at 10:44 PM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:
>
>That's right Andy. 0.22+. We are running a HDFS-RAID code base that is pretty close to what is available in Apache hdfs trunk.
>>
>>
>>-dhruba
>>
>>
>>
>>On Thu, Sep 15, 2011 at 10:08 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>>
>>But that is the HDFS RAID effectively in 0.22+, not 0.21, right Dhruba?
>>>
>>> 
>>>Best regards,
>>>
>>>
>>>       - Andy
>>>
>>>Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>>>
>>>
>>>>________________________________
>>>>From: Dhruba Borthakur <[EMAIL PROTECTED]>
>>>>To: [EMAIL PROTECTED]
>>>>Sent: Thursday, September 15, 2011 10:06 AM
>>>>Subject: Re: Need help regarding HDFS-RAID
>>>>
>>>>
>>>>
>>>>We use HDFS RAID in a big way. Data older than 12 days are RAIDED using XOR encoding (effective replication of 2.5). Data older than a few months are raided using ReedSolomon (effective observed replication factor of 1.5). This is running on our 60 PB size cluster for about an year now.
>>>>
>>>>
>>>>thanks
>>>>dhruba
>>>>
>>>>
>>>>
>>>>On Thu, Sep 15, 2011 at 5:31 AM, Ajit Ratnaparkhi <[EMAIL PROTECTED]> wrote:
>>>>
>>>>Hi,
>>>>>
>>>>>
>>>>>We were planning to use it for past data archival(instead of moving it to archival store).
>>>>>Archiving it in HDFS gives advantage of making it easily available for processing whenever required.
>>>>>
>>>>>
>>>>>Is there any archival solution in hadoop ecosystem?
>>>>>
>>>>>
>>>>>thanks,
>>>>>Ajit.
>>>>>
>>>>>
>>>>>
>>>>>On Thu, Sep 15, 2011 at 5:05 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>Hey Ajit,
>>>>>>
>>>>>>HDFS-RAID was never part of the 0.20 release. It made its debut in the
>>>>>>0.21 release [1]. I know that Facebook uses it (and also did develop
>>>>>>it), but unsure of users beyond Facebook.
>>>>>>
>>>>>>While 0.21 overall is not entirely deemed as production-usable yet
>>>>>>(and is in fact, possibly abandoned for efforts on 0.22+), you can
>>>>>>give that release a whirl on a test cluster and see for yourself if
>>>>>>your need beats the stability.
>>>>>>
>>>>>>Just curious though - why are you looking to use this specifically?
>>>>>>
>>>>>>[1] - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21/mapreduce/src/contrib/raid/
>>>>>>
>>>>>>
>>>>>>On Thu, Sep 15, 2011 at 4:37 PM, Ajit Ratnaparkhi
>>>>>><[EMAIL PROTECTED]> wrote:
>>>>>>> Hi,
>>>>>>> We want to use HDFS-RAID in our production cluster.
>>>>>>> (http://wiki.apache.org/hadoop/HDFS-RAID)
>>>>>>> I am not able to find source/binaries/configs for this in official hadoop
>>>>>>> distribution from apache hadoop. (checked in 0.20.1 and 0.20.2).
>>>>>>> Can somebody please tell me where can I find that? and installation
>>>>>>> procedure?
>>>>>>> Also, is HDFS-RAID implementation stable enough to use in production?
>>>>>>> thanks,
>>>>>>> Ajit.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>--
>>>>>>Harsh J
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>--
>>>>Connect to me at http://www.facebook.com/dhruba
>>>>
>>>>
>>>>
>>
>>
>>
>>--
>>Connect to me at http://www.facebook.com/dhruba
>>
>
>
>
+
Ajit Ratnaparkhi 2011-09-16, 05:43
+
Andrew Purtell 2011-09-17, 16:16
+
Dhruba Borthakur 2011-09-20, 09:18
+
Ajit Ratnaparkhi 2011-09-20, 13:49
+
Andrew Purtell 2011-09-20, 16:03
+
Dhruba Borthakur 2011-09-20, 16:49
+
Andrew Purtell 2011-09-20, 23:10
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB