Nice explanation guys .. thanks

Syed Abdul kather
send from Samsung S3
On Aug 9, 2012 12:02 AM, "Ajit Ratnaparkhi [via Lucene]" <
ml-node+[EMAIL PROTECTED]> wrote:

> Agreed with Steve.
> That is most important use of HDFS RAID, where you consume less disk space
> with same reliability and availability guarantee at cost of processing
> performance. Most of data in hdfs is cold data, without HDFS RAID you end
> up maintaining 3 replicas of data which is hardly going to be processed
> again, but you cant remove/move this data to separate archive because if
>  required processing should be as soon as possible.
>
> -Ajit
>
> On Wed, Aug 8, 2012 at 11:01 PM, Steve Loughran <[hidden email]<http://user/SendEmail.jtp?type=node&node=3999922&i=0>
> > wrote:
>
>>
>>
>> On 8 August 2012 09:46, Sourygna Luangsay <[hidden email]<http://user/SendEmail.jtp?type=node&node=3999922&i=1>
>> > wrote:
>>
>>>  Hi folks!****
>>>
>>> One of the scenario I can think in order to take advantage of HDFS RAID
>>> without suffering this penalty is:**
>>>
>>> **-          **Using normal HDFS with default replication=3 for my
>>> “fresh data”****
>>>
>>> **-          **Using HDFS RAID for my historical data (that is barely
>>> used by M/R)****
>>>
>>> ** **
>>>
>>>
>>>
>> exactly: less space use on cold data, with the penalty that access
>> performance can be worse. As the majority of data on a hadoop cluster is
>> usually "cold", it's a space and power efficient story for the archive data
>>
>> --
>> Steve Loughran
>> Hortonworks Inc
>>
>>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/is-HDFS-RAID-data-locality-efficient-tp3999891p3999922.html
>  To unsubscribe from Lucene, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=472066&code=aW4uYWJkdWxAZ21haWwuY29tfDQ3MjA2NnwxMDczOTUyNDEw>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
-----
THANKS AND REGARDS,
SYED ABDUL KATHER
--
View this message in context: http://lucene.472066.n3.nabble.com/is-HDFS-RAID-data-locality-efficient-tp3999891p3999924.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Ok...

So under Apache Hadoop, how do you specify the location of when and where a directory will be created on HDFS?

As an example, if I want to create a /coldData directory in HDFS as a place to store my older data sets, How does that get assigned specifically to a RAIDed HDFS?
(Or even specific machines?)

I know I can do this in MapR's distribution, but I am not aware of this feature being made available in the Apache based releases?

Is this part of the latest feature set?

Thx

-Mike

On Aug 8, 2012, at 12:31 PM, Steve Loughran <[EMAIL PROTECTED]> wrote:

>
>
> On 8 August 2012 09:46, Sourygna Luangsay <[EMAIL PROTECTED]> wrote:
> Hi folks!
>
> One of the scenario I can think in order to take advantage of HDFS RAID without suffering this penalty is:
>
> -          Using normal HDFS with default replication=3 for my “fresh data”
>
> -          Using HDFS RAID for my historical data (that is barely used by M/R)
>
>  
>
>
>
>
> exactly: less space use on cold data, with the penalty that access performance can be worse. As the majority of data on a hadoop cluster is usually "cold", it's a space and power efficient story for the archive data
>
> --
> Steve Loughran
> Hortonworks Inc
>

Hi,

 

Thanks a lot everybody for your replies.

Your ideas about cold and hot data using different storage policies prove to
be very interesting.

 

Regards,

 

Sourygna Luangsay