Hi,

What is an acceptable CPU_WIO % while running an heavy MR job? Should
we also try to keep that under 10%? Or it's not realistic and we will
see more about 50%?

One of my nodes is showing 70% :( It's WAY to much. I will add another
disk tomorrow and put them in RAID0, but I'm wondering how low shoud I
go?

JM

Hey JM,

  Why RAID0?  That has a lot of disadvantages to using a JBOD
configuration?  Wait I/O is a symptom, not a problem.  Are you actually
experiencing a problem or are you treating for something you think should
be lower?

On Thu, Feb 7, 2013 at 8:19 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> What is an acceptable CPU_WIO % while running an heavy MR job? Should
> we also try to keep that under 10%? Or it's not realistic and we will
> see more about 50%?
>
> One of my nodes is showing 70% :( It's WAY to much. I will add another
> disk tomorrow and put them in RAID0, but I'm wondering how low shoud I
> go?
>
> JM
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

Hi Kevin,

I'm facing some issues on one of my nodes and I'm trying to find a way
to fix that. CPU is used about 10% by user, and 80% for WIO. So I'm
looking for a way to improve that. The mother board can do RAIDx and
JBOD too. It's the server I used few weeks ago to run some disks
benchs.

http://www.spaggiari.org/index.php/hbase/hard-drives-performances

The conclusion was that RAID0 was 70% faster than JBOD. But JBOD was
faster than RAID1.

I have a 2TB drive in this server and was thinking about just adding
another 2TB drive.

What are the advantages of JBOD compared to RAID0? From the last tests
I did, it was slower.

Since I will have to re-format the disks anyway, I can re-run the
tests just in case I did not configured something properly....

JM

2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> Hey JM,
>
>   Why RAID0?  That has a lot of disadvantages to using a JBOD
> configuration?  Wait I/O is a symptom, not a problem.  Are you actually
> experiencing a problem or are you treating for something you think should
> be lower?
>
> On Thu, Feb 7, 2013 at 8:19 PM, Jean-Marc Spaggiari
> <[EMAIL PROTECTED]
>> wrote:
>
>> Hi,
>>
>> What is an acceptable CPU_WIO % while running an heavy MR job? Should
>> we also try to keep that under 10%? Or it's not realistic and we will
>> see more about 50%?
>>
>> One of my nodes is showing 70% :( It's WAY to much. I will add another
>> disk tomorrow and put them in RAID0, but I'm wondering how low shoud I
>> go?
>>
>> JM
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

JM,

  Okay, I think I see what was happening.  You currently only have one
drive in the system that is showing High I/O wait correct?  You are looking
at bringing in a second drive to help distribute the load?  In your testing
with two drives you saw that RAID0 offerred superior performance vs JBOD.
 Typically when we see RAID vs JBOD we are dealing with about 6 - 12
drives.  Here are some of the pluses and minuses:

RAID0 - faster performance since the data is striped, but you are as fast
as your slowest drive and one drive failure you lose the whole volume.

JBOD - Better redundancy and faster than a RAID1, or a RAID5
configuration(unsure about a RAID4), but you are slower than RAID0

It sounds like since you only have 1 drive in the node right now, you
wouldn't be gaining or losing any redundancy by going with RAID0.  For what
it is worth, I would agree that you are I/O bound.  If you run a sar -A >
/tmp/sar.out and you take a look at the drive utilization what is your
TPS(IOPs) count that you are seeing?

On Thu, Feb 7, 2013 at 9:00 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> Hi Kevin,
>
> I'm facing some issues on one of my nodes and I'm trying to find a way
> to fix that. CPU is used about 10% by user, and 80% for WIO. So I'm
> looking for a way to improve that. The mother board can do RAIDx and
> JBOD too. It's the server I used few weeks ago to run some disks
> benchs.
>
> http://www.spaggiari.org/index.php/hbase/hard-drives-performances
>
> The conclusion was that RAID0 was 70% faster than JBOD. But JBOD was
> faster than RAID1.
>
> I have a 2TB drive in this server and was thinking about just adding
> another 2TB drive.
>
> What are the advantages of JBOD compared to RAID0? From the last tests
> I did, it was slower.
>
> Since I will have to re-format the disks anyway, I can re-run the
> tests just in case I did not configured something properly....
>
> JM
>
> 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> > Hey JM,
> >
> >   Why RAID0?  That has a lot of disadvantages to using a JBOD
> > configuration?  Wait I/O is a symptom, not a problem.  Are you actually
> > experiencing a problem or are you treating for something you think should
> > be lower?
> >
> > On Thu, Feb 7, 2013 at 8:19 PM, Jean-Marc Spaggiari
> > <[EMAIL PROTECTED]
> >> wrote:
> >
> >> Hi,
> >>
> >> What is an acceptable CPU_WIO % while running an heavy MR job? Should
> >> we also try to keep that under 10%? Or it's not realistic and we will
> >> see more about 50%?
> >>
> >> One of my nodes is showing 70% :( It's WAY to much. I will add another
> >> disk tomorrow and put them in RAID0, but I'm wondering how low shoud I
> >> go?
> >>
> >> JM
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
> >
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera

Ok. I see with RAID0 might be better for me compare to JBOD. Also, why
do we want to use RAID1 or RAID5? We already have the redundancy done
by hadoop, is it not going to add another non-required level of
redundancy?

Should I already think to have 3 or even 4 drives in each node?

I tried sar -A and it's only giving me 2 lines.
root@node7:/home/hbase# sar -A
Linux 3.2.0-4-amd64 (node7) 2013-02-07 _x86_64_ (4 CPU)

21:29:54          LINUX RESTART

It was not enabled, so I just enabled it and restart sysstat, but
seems that it's still not populated.

I have the diskstats plugin installed on ganglia, so I have a LOT of
disks information, but not this specific one.

My write_bytes_per_sec is pretty low. Average is 232K for the last 2
hours. But my erad_bytes_per_sec is avera 22.83M for the same period.
The graph is looking like a comb.

I just retried sar and some data is coming.. I will need to let it run
for few more minutes to get some more data ...

JM
2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> JM,
>
>   Okay, I think I see what was happening.  You currently only have one
> drive in the system that is showing High I/O wait correct?  You are looking
> at bringing in a second drive to help distribute the load?  In your testing
> with two drives you saw that RAID0 offerred superior performance vs JBOD.
>  Typically when we see RAID vs JBOD we are dealing with about 6 - 12
> drives.  Here are some of the pluses and minuses:
>
> RAID0 - faster performance since the data is striped, but you are as fast
> as your slowest drive and one drive failure you lose the whole volume.
>
> JBOD - Better redundancy and faster than a RAID1, or a RAID5
> configuration(unsure about a RAID4), but you are slower than RAID0
>
> It sounds like since you only have 1 drive in the node right now, you
> wouldn't be gaining or losing any redundancy by going with RAID0.  For what
> it is worth, I would agree that you are I/O bound.  If you run a sar -A >
> /tmp/sar.out and you take a look at the drive utilization what is your
> TPS(IOPs) count that you are seeing?
>
> On Thu, Feb 7, 2013 at 9:00 PM, Jean-Marc Spaggiari
> <[EMAIL PROTECTED]
>> wrote:
>
>> Hi Kevin,
>>
>> I'm facing some issues on one of my nodes and I'm trying to find a way
>> to fix that. CPU is used about 10% by user, and 80% for WIO. So I'm
>> looking for a way to improve that. The mother board can do RAIDx and
>> JBOD too. It's the server I used few weeks ago to run some disks
>> benchs.
>>
>> http://www.spaggiari.org/index.php/hbase/hard-drives-performances
>>
>> The conclusion was that RAID0 was 70% faster than JBOD. But JBOD was
>> faster than RAID1.
>>
>> I have a 2TB drive in this server and was thinking about just adding
>> another 2TB drive.
>>
>> What are the advantages of JBOD compared to RAID0? From the last tests
>> I did, it was slower.
>>
>> Since I will have to re-format the disks anyway, I can re-run the
>> tests just in case I did not configured something properly....
>>
>> JM
>>
>> 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
>> > Hey JM,
>> >
>> >   Why RAID0?  That has a lot of disadvantages to using a JBOD
>> > configuration?  Wait I/O is a symptom, not a problem.  Are you actually
>> > experiencing a problem or are you treating for something you think
>> > should
>> > be lower?
>> >
>> > On Thu, Feb 7, 2013 at 8:19 PM, Jean-Marc Spaggiari
>> > <[EMAIL PROTECTED]
>> >> wrote:
>> >
>> >> Hi,
>> >>
>> >> What is an acceptable CPU_WIO % while running an heavy MR job? Should
>> >> we also try to keep that under 10%? Or it's not realistic and we will
>> >> see more about 50%?
>> >>
>> >> One of my nodes is showing 70% :( It's WAY to much. I will add another
>> >> disk tomorrow and put them in RAID0, but I'm wondering how low shoud I
>> >> go?
>> >>
>> >> JM
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>> >
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

JM,

  I think you misunderstood me.  I am not advocating any form of RAID for
Hadoop.  It is true that we already have redundancy built in with HDFS.  So
unless you were going to do something silly like sacrifice speed to run
RAID1 or RAID5 and lower your replication to 2...just don't do it :)
 Anyway, yes you probably should have 3 - 4 drives per node if not more.
 At that point then the you will really see the benefit of JBOD over RAID0

Do you want to be able to lose a drive and keep the node up?  If yes, then
JBOD is for you.  Do you not care if you lose that node due to drive
failure? You just need speed, then RAID0 may be the correct choice.  Sar
will take some time to populate.  Give it about 24 hours and you should be
able to glean some interesting information.

On Thu, Feb 7, 2013 at 9:50 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> Ok. I see with RAID0 might be better for me compare to JBOD. Also, why
> do we want to use RAID1 or RAID5? We already have the redundancy done
> by hadoop, is it not going to add another non-required level of
> redundancy?
>
> Should I already think to have 3 or even 4 drives in each node?
>
> I tried sar -A and it's only giving me 2 lines.
> root@node7:/home/hbase# sar -A
> Linux 3.2.0-4-amd64 (node7)     2013-02-07      _x86_64_        (4 CPU)
>
> 21:29:54          LINUX RESTART
>
> It was not enabled, so I just enabled it and restart sysstat, but
> seems that it's still not populated.
>
> I have the diskstats plugin installed on ganglia, so I have a LOT of
> disks information, but not this specific one.
>
> My write_bytes_per_sec is pretty low. Average is 232K for the last 2
> hours. But my erad_bytes_per_sec is avera 22.83M for the same period.
> The graph is looking like a comb.
>
> I just retried sar and some data is coming.. I will need to let it run
> for few more minutes to get some more data ...
>
> JM
>
>
> 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> > JM,
> >
> >   Okay, I think I see what was happening.  You currently only have one
> > drive in the system that is showing High I/O wait correct?  You are
> looking
> > at bringing in a second drive to help distribute the load?  In your
> testing
> > with two drives you saw that RAID0 offerred superior performance vs JBOD.
> >  Typically when we see RAID vs JBOD we are dealing with about 6 - 12
> > drives.  Here are some of the pluses and minuses:
> >
> > RAID0 - faster performance since the data is striped, but you are as fast
> > as your slowest drive and one drive failure you lose the whole volume.
> >
> > JBOD - Better redundancy and faster than a RAID1, or a RAID5
> > configuration(unsure about a RAID4), but you are slower than RAID0
> >
> > It sounds like since you only have 1 drive in the node right now, you
> > wouldn't be gaining or losing any redundancy by going with RAID0.  For
> what
> > it is worth, I would agree that you are I/O bound.  If you run a sar -A >
> > /tmp/sar.out and you take a look at the drive utilization what is your
> > TPS(IOPs) count that you are seeing?
> >
> > On Thu, Feb 7, 2013 at 9:00 PM, Jean-Marc Spaggiari
> > <[EMAIL PROTECTED]
> >> wrote:
> >
> >> Hi Kevin,
> >>
> >> I'm facing some issues on one of my nodes and I'm trying to find a way
> >> to fix that. CPU is used about 10% by user, and 80% for WIO. So I'm
> >> looking for a way to improve that. The mother board can do RAIDx and
> >> JBOD too. It's the server I used few weeks ago to run some disks
> >> benchs.
> >>
> >> http://www.spaggiari.org/index.php/hbase/hard-drives-performances
> >>
> >> The conclusion was that RAID0 was 70% faster than JBOD. But JBOD was
> >> faster than RAID1.
> >>
> >> I have a 2TB drive in this server and was thinking about just adding
> >> another 2TB drive.
> >>
> >> What are the advantages of JBOD compared to RAID0? From the last tests
> >> I did, it was slower.
> >>
> >> Since I will have to re-format the disks anyway, I can re-run the
> >> tests just in case I did not configured something properly....

Kevin O'Dell
Customer Operations Engineer, Cloudera

Ok. I see. For my usecase I prefer to loose the data and have faster
process. So I will go for RAID0 and keep the replication factor to
3... If at some point I have 5 disks in the node, I will most probably
give a try to RAID5 and see the performances compared to the other
RAID/JBOD options.

Is there a "rule", like, 1 HD per core? Or we can't really simplify that much?

So far I have that in the sar output:
21:35:03          tps      rtps      wtps   bread/s   bwrtn/s
21:45:03       218,85    215,97      2,88  45441,95    308,04
21:55:02       209,73    206,67      3,06  43985,28    378,32
22:05:04       215,03    211,71      3,33  44831,00    312,95
Average :      214,54    211,45      3,09  44753,09    333,07

But I'm not sure what it means. I will wait for tomorrow to get more
results, but my job will be done over night, so I'm not sure the
average will be accurate...

JM
2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> JM,
>
>   I think you misunderstood me.  I am not advocating any form of RAID for
> Hadoop.  It is true that we already have redundancy built in with HDFS.  So
> unless you were going to do something silly like sacrifice speed to run
> RAID1 or RAID5 and lower your replication to 2...just don't do it :)
>  Anyway, yes you probably should have 3 - 4 drives per node if not more.
>  At that point then the you will really see the benefit of JBOD over RAID0
>
> Do you want to be able to lose a drive and keep the node up?  If yes, then
> JBOD is for you.  Do you not care if you lose that node due to drive
> failure? You just need speed, then RAID0 may be the correct choice.  Sar
> will take some time to populate.  Give it about 24 hours and you should be
> able to glean some interesting information.
>
> On Thu, Feb 7, 2013 at 9:50 PM, Jean-Marc Spaggiari
> <[EMAIL PROTECTED]
>> wrote:
>
>> Ok. I see with RAID0 might be better for me compare to JBOD. Also, why
>> do we want to use RAID1 or RAID5? We already have the redundancy done
>> by hadoop, is it not going to add another non-required level of
>> redundancy?
>>
>> Should I already think to have 3 or even 4 drives in each node?
>>
>> I tried sar -A and it's only giving me 2 lines.
>> root@node7:/home/hbase# sar -A
>> Linux 3.2.0-4-amd64 (node7)     2013-02-07      _x86_64_        (4 CPU)
>>
>> 21:29:54          LINUX RESTART
>>
>> It was not enabled, so I just enabled it and restart sysstat, but
>> seems that it's still not populated.
>>
>> I have the diskstats plugin installed on ganglia, so I have a LOT of
>> disks information, but not this specific one.
>>
>> My write_bytes_per_sec is pretty low. Average is 232K for the last 2
>> hours. But my erad_bytes_per_sec is avera 22.83M for the same period.
>> The graph is looking like a comb.
>>
>> I just retried sar and some data is coming.. I will need to let it run
>> for few more minutes to get some more data ...
>>
>> JM
>>
>>
>> 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
>> > JM,
>> >
>> >   Okay, I think I see what was happening.  You currently only have one
>> > drive in the system that is showing High I/O wait correct?  You are
>> looking
>> > at bringing in a second drive to help distribute the load?  In your
>> testing
>> > with two drives you saw that RAID0 offerred superior performance vs
>> > JBOD.
>> >  Typically when we see RAID vs JBOD we are dealing with about 6 - 12
>> > drives.  Here are some of the pluses and minuses:
>> >
>> > RAID0 - faster performance since the data is striped, but you are as
>> > fast
>> > as your slowest drive and one drive failure you lose the whole volume.
>> >
>> > JBOD - Better redundancy and faster than a RAID1, or a RAID5
>> > configuration(unsure about a RAID4), but you are slower than RAID0
>> >
>> > It sounds like since you only have 1 drive in the node right now, you
>> > wouldn't be gaining or losing any redundancy by going with RAID0.  For
>> what
>> > it is worth, I would agree that you are I/O bound.  If you run a sar -A
>> > >
>> > /tmp/sar.out and you take a look at the drive utilization what is your

JM，

I don't have the context, but if you are using Hadoop/Hbase, so don't do
RAID on your disk.
On Fri, Feb 8, 2013 at 11:15 AM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Ok. I see. For my usecase I prefer to loose the data and have faster
> process. So I will go for RAID0 and keep the replication factor to
> 3... If at some point I have 5 disks in the node, I will most probably
> give a try to RAID5 and see the performances compared to the other
> RAID/JBOD options.
>
> Is there a "rule", like, 1 HD per core? Or we can't really simplify that
> much?
>
> So far I have that in the sar output:
> 21:35:03          tps      rtps      wtps   bread/s   bwrtn/s
> 21:45:03       218,85    215,97      2,88  45441,95    308,04
> 21:55:02       209,73    206,67      3,06  43985,28    378,32
> 22:05:04       215,03    211,71      3,33  44831,00    312,95
> Average :      214,54    211,45      3,09  44753,09    333,07
>
> But I'm not sure what it means. I will wait for tomorrow to get more
> results, but my job will be done over night, so I'm not sure the
> average will be accurate...
>
> JM
>
>
> 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> > JM,
> >
> >   I think you misunderstood me.  I am not advocating any form of RAID for
> > Hadoop.  It is true that we already have redundancy built in with HDFS.
>  So
> > unless you were going to do something silly like sacrifice speed to run
> > RAID1 or RAID5 and lower your replication to 2...just don't do it :)
> >  Anyway, yes you probably should have 3 - 4 drives per node if not more.
> >  At that point then the you will really see the benefit of JBOD over
> RAID0
> >
> > Do you want to be able to lose a drive and keep the node up?  If yes,
> then
> > JBOD is for you.  Do you not care if you lose that node due to drive
> > failure? You just need speed, then RAID0 may be the correct choice.  Sar
> > will take some time to populate.  Give it about 24 hours and you should
> be
> > able to glean some interesting information.
> >
> > On Thu, Feb 7, 2013 at 9:50 PM, Jean-Marc Spaggiari
> > <[EMAIL PROTECTED]
> >> wrote:
> >
> >> Ok. I see with RAID0 might be better for me compare to JBOD. Also, why
> >> do we want to use RAID1 or RAID5? We already have the redundancy done
> >> by hadoop, is it not going to add another non-required level of
> >> redundancy?
> >>
> >> Should I already think to have 3 or even 4 drives in each node?
> >>
> >> I tried sar -A and it's only giving me 2 lines.
> >> root@node7:/home/hbase# sar -A
> >> Linux 3.2.0-4-amd64 (node7)     2013-02-07      _x86_64_        (4 CPU)
> >>
> >> 21:29:54          LINUX RESTART
> >>
> >> It was not enabled, so I just enabled it and restart sysstat, but
> >> seems that it's still not populated.
> >>
> >> I have the diskstats plugin installed on ganglia, so I have a LOT of
> >> disks information, but not this specific one.
> >>
> >> My write_bytes_per_sec is pretty low. Average is 232K for the last 2
> >> hours. But my erad_bytes_per_sec is avera 22.83M for the same period.
> >> The graph is looking like a comb.
> >>
> >> I just retried sar and some data is coming.. I will need to let it run
> >> for few more minutes to get some more data ...
> >>
> >> JM
> >>
> >>
> >> 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> >> > JM,
> >> >
> >> >   Okay, I think I see what was happening.  You currently only have one
> >> > drive in the system that is showing High I/O wait correct?  You are
> >> looking
> >> > at bringing in a second drive to help distribute the load?  In your
> >> testing
> >> > with two drives you saw that RAID0 offerred superior performance vs
> >> > JBOD.
> >> >  Typically when we see RAID vs JBOD we are dealing with about 6 - 12
> >> > drives.  Here are some of the pluses and minuses:
> >> >
> >> > RAID0 - faster performance since the data is striped, but you are as
> >> > fast
> >> > as your slowest drive and one drive failure you lose the whole volume.
> >> >
> >> > JBOD - Better redundancy and faster than a RAID1, or a RAID5

Azuryy,

  The main reason to recommend against RAID is that it is slow and it adds
redundancy that we already have in Hadoop.  RAID0 is another story as long
as all of the drives are healthy and you don't mind losing the whole volume
if you lose one drive.

JM,

  I would not even waste my time testing RAID5 or RAID6(unless it is just
for educational purposes :) ).  200+ IOPs consistently on one SATA drive is
pretty high, that would explain your high I/O wait time.  If your use case
allows for you to lose the whole node, there is not a good reason for you
to shy away from RAID0.  Please let us know how this plays out with your
environment.

On Thu, Feb 7, 2013 at 10:23 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:

> JM，
>
> I don't have the context, but if you are using Hadoop/Hbase, so don't do
> RAID on your disk.
>
>
> On Fri, Feb 8, 2013 at 11:15 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > Ok. I see. For my usecase I prefer to loose the data and have faster
> > process. So I will go for RAID0 and keep the replication factor to
> > 3... If at some point I have 5 disks in the node, I will most probably
> > give a try to RAID5 and see the performances compared to the other
> > RAID/JBOD options.
> >
> > Is there a "rule", like, 1 HD per core? Or we can't really simplify that
> > much?
> >
> > So far I have that in the sar output:
> > 21:35:03          tps      rtps      wtps   bread/s   bwrtn/s
> > 21:45:03       218,85    215,97      2,88  45441,95    308,04
> > 21:55:02       209,73    206,67      3,06  43985,28    378,32
> > 22:05:04       215,03    211,71      3,33  44831,00    312,95
> > Average :      214,54    211,45      3,09  44753,09    333,07
> >
> > But I'm not sure what it means. I will wait for tomorrow to get more
> > results, but my job will be done over night, so I'm not sure the
> > average will be accurate...
> >
> > JM
> >
> >
> > 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> > > JM,
> > >
> > >   I think you misunderstood me.  I am not advocating any form of RAID
> for
> > > Hadoop.  It is true that we already have redundancy built in with HDFS.
> >  So
> > > unless you were going to do something silly like sacrifice speed to run
> > > RAID1 or RAID5 and lower your replication to 2...just don't do it :)
> > >  Anyway, yes you probably should have 3 - 4 drives per node if not
> more.
> > >  At that point then the you will really see the benefit of JBOD over
> > RAID0
> > >
> > > Do you want to be able to lose a drive and keep the node up?  If yes,
> > then
> > > JBOD is for you.  Do you not care if you lose that node due to drive
> > > failure? You just need speed, then RAID0 may be the correct choice.
>  Sar
> > > will take some time to populate.  Give it about 24 hours and you should
> > be
> > > able to glean some interesting information.
> > >
> > > On Thu, Feb 7, 2013 at 9:50 PM, Jean-Marc Spaggiari
> > > <[EMAIL PROTECTED]
> > >> wrote:
> > >
> > >> Ok. I see with RAID0 might be better for me compare to JBOD. Also, why
> > >> do we want to use RAID1 or RAID5? We already have the redundancy done
> > >> by hadoop, is it not going to add another non-required level of
> > >> redundancy?
> > >>
> > >> Should I already think to have 3 or even 4 drives in each node?
> > >>
> > >> I tried sar -A and it's only giving me 2 lines.
> > >> root@node7:/home/hbase# sar -A
> > >> Linux 3.2.0-4-amd64 (node7)     2013-02-07      _x86_64_        (4
> CPU)
> > >>
> > >> 21:29:54          LINUX RESTART
> > >>
> > >> It was not enabled, so I just enabled it and restart sysstat, but
> > >> seems that it's still not populated.
> > >>
> > >> I have the diskstats plugin installed on ganglia, so I have a LOT of
> > >> disks information, but not this specific one.
> > >>
> > >> My write_bytes_per_sec is pretty low. Average is 232K for the last 2
> > >> hours. But my erad_bytes_per_sec is avera 22.83M for the same period.
> > >> The graph is looking like a comb.
> > >>
> > >> I just retried sar and some data is coming.. I will need to let it run

Kevin O'Dell
Customer Operations Engineer, Cloudera

Hi Kevin,

I think it will take time before I get a chance to have 5 drives in
the same server, so I will see at that time to test RAID5.

I'm going to add one drive per server today or tomorrow to try to
improve that. What IOPs should I try to have? 100? Less? It will all
be SATA3 drives and I will configure all in RAID0.

It doesn't seems to me to be an issue to lose one node, since data
will be replicated everywhere else. I will "simply" have to replace
the failing disk and restart the node, no?

JM

2013/2/8, Kevin O'dell <[EMAIL PROTECTED]>:
> Azuryy,
>
>   The main reason to recommend against RAID is that it is slow and it adds
> redundancy that we already have in Hadoop.  RAID0 is another story as long
> as all of the drives are healthy and you don't mind losing the whole volume
> if you lose one drive.
>
> JM,
>
>   I would not even waste my time testing RAID5 or RAID6(unless it is just
> for educational purposes :) ).  200+ IOPs consistently on one SATA drive is
> pretty high, that would explain your high I/O wait time.  If your use case
> allows for you to lose the whole node, there is not a good reason for you
> to shy away from RAID0.  Please let us know how this plays out with your
> environment.
>
> On Thu, Feb 7, 2013 at 10:23 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>
>> JM，
>>
>> I don't have the context, but if you are using Hadoop/Hbase, so don't do
>> RAID on your disk.
>>
>>
>> On Fri, Feb 8, 2013 at 11:15 AM, Jean-Marc Spaggiari <
>> [EMAIL PROTECTED]> wrote:
>>
>> > Ok. I see. For my usecase I prefer to loose the data and have faster
>> > process. So I will go for RAID0 and keep the replication factor to
>> > 3... If at some point I have 5 disks in the node, I will most probably
>> > give a try to RAID5 and see the performances compared to the other
>> > RAID/JBOD options.
>> >
>> > Is there a "rule", like, 1 HD per core? Or we can't really simplify
>> > that
>> > much?
>> >
>> > So far I have that in the sar output:
>> > 21:35:03          tps      rtps      wtps   bread/s   bwrtn/s
>> > 21:45:03       218,85    215,97      2,88  45441,95    308,04
>> > 21:55:02       209,73    206,67      3,06  43985,28    378,32
>> > 22:05:04       215,03    211,71      3,33  44831,00    312,95
>> > Average :      214,54    211,45      3,09  44753,09    333,07
>> >
>> > But I'm not sure what it means. I will wait for tomorrow to get more
>> > results, but my job will be done over night, so I'm not sure the
>> > average will be accurate...
>> >
>> > JM
>> >
>> >
>> > 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
>> > > JM,
>> > >
>> > >   I think you misunderstood me.  I am not advocating any form of RAID
>> for
>> > > Hadoop.  It is true that we already have redundancy built in with
>> > > HDFS.
>> >  So
>> > > unless you were going to do something silly like sacrifice speed to
>> > > run
>> > > RAID1 or RAID5 and lower your replication to 2...just don't do it :)
>> > >  Anyway, yes you probably should have 3 - 4 drives per node if not
>> more.
>> > >  At that point then the you will really see the benefit of JBOD over
>> > RAID0
>> > >
>> > > Do you want to be able to lose a drive and keep the node up?  If yes,
>> > then
>> > > JBOD is for you.  Do you not care if you lose that node due to drive
>> > > failure? You just need speed, then RAID0 may be the correct choice.
>>  Sar
>> > > will take some time to populate.  Give it about 24 hours and you
>> > > should
>> > be
>> > > able to glean some interesting information.
>> > >
>> > > On Thu, Feb 7, 2013 at 9:50 PM, Jean-Marc Spaggiari
>> > > <[EMAIL PROTECTED]
>> > >> wrote:
>> > >
>> > >> Ok. I see with RAID0 might be better for me compare to JBOD. Also,
>> > >> why
>> > >> do we want to use RAID1 or RAID5? We already have the redundancy
>> > >> done
>> > >> by hadoop, is it not going to add another non-required level of
>> > >> redundancy?
>> > >>
>> > >> Should I already think to have 3 or even 4 drives in each node?
>> > >>
>> > >> I tried sar -A and it's only giving me 2 lines.

JM,

  Basically, you will have to replace failed disk and rebuild RAID0 since
the other half of the data is worthless.  There is not a real recommended
value, but anything under 150 - 200 would make me more comfortable.

On Fri, Feb 8, 2013 at 10:43 AM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi Kevin,
>
> I think it will take time before I get a chance to have 5 drives in
> the same server, so I will see at that time to test RAID5.
>
> I'm going to add one drive per server today or tomorrow to try to
> improve that. What IOPs should I try to have? 100? Less? It will all
> be SATA3 drives and I will configure all in RAID0.
>
> It doesn't seems to me to be an issue to lose one node, since data
> will be replicated everywhere else. I will "simply" have to replace
> the failing disk and restart the node, no?
>
> JM
>
> 2013/2/8, Kevin O'dell <[EMAIL PROTECTED]>:
> > Azuryy,
> >
> >   The main reason to recommend against RAID is that it is slow and it
> adds
> > redundancy that we already have in Hadoop.  RAID0 is another story as
> long
> > as all of the drives are healthy and you don't mind losing the whole
> volume
> > if you lose one drive.
> >
> > JM,
> >
> >   I would not even waste my time testing RAID5 or RAID6(unless it is just
> > for educational purposes :) ).  200+ IOPs consistently on one SATA drive
> is
> > pretty high, that would explain your high I/O wait time.  If your use
> case
> > allows for you to lose the whole node, there is not a good reason for you
> > to shy away from RAID0.  Please let us know how this plays out with your
> > environment.
> >
> > On Thu, Feb 7, 2013 at 10:23 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
> >
> >> JM，
> >>
> >> I don't have the context, but if you are using Hadoop/Hbase, so don't do
> >> RAID on your disk.
> >>
> >>
> >> On Fri, Feb 8, 2013 at 11:15 AM, Jean-Marc Spaggiari <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > Ok. I see. For my usecase I prefer to loose the data and have faster
> >> > process. So I will go for RAID0 and keep the replication factor to
> >> > 3... If at some point I have 5 disks in the node, I will most probably
> >> > give a try to RAID5 and see the performances compared to the other
> >> > RAID/JBOD options.
> >> >
> >> > Is there a "rule", like, 1 HD per core? Or we can't really simplify
> >> > that
> >> > much?
> >> >
> >> > So far I have that in the sar output:
> >> > 21:35:03          tps      rtps      wtps   bread/s   bwrtn/s
> >> > 21:45:03       218,85    215,97      2,88  45441,95    308,04
> >> > 21:55:02       209,73    206,67      3,06  43985,28    378,32
> >> > 22:05:04       215,03    211,71      3,33  44831,00    312,95
> >> > Average :      214,54    211,45      3,09  44753,09    333,07
> >> >
> >> > But I'm not sure what it means. I will wait for tomorrow to get more
> >> > results, but my job will be done over night, so I'm not sure the
> >> > average will be accurate...
> >> >
> >> > JM
> >> >
> >> >
> >> > 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
> >> > > JM,
> >> > >
> >> > >   I think you misunderstood me.  I am not advocating any form of
> RAID
> >> for
> >> > > Hadoop.  It is true that we already have redundancy built in with
> >> > > HDFS.
> >> >  So
> >> > > unless you were going to do something silly like sacrifice speed to
> >> > > run
> >> > > RAID1 or RAID5 and lower your replication to 2...just don't do it :)
> >> > >  Anyway, yes you probably should have 3 - 4 drives per node if not
> >> more.
> >> > >  At that point then the you will really see the benefit of JBOD over
> >> > RAID0
> >> > >
> >> > > Do you want to be able to lose a drive and keep the node up?  If
> yes,
> >> > then
> >> > > JBOD is for you.  Do you not care if you lose that node due to drive
> >> > > failure? You just need speed, then RAID0 may be the correct choice.
> >>  Sar
> >> > > will take some time to populate.  Give it about 24 hours and you
> >> > > should
> >> > be
> >> > > able to glean some interesting information.

Kevin O'Dell
Customer Operations Engineer, Cloudera

Perfect, thanks Kevin.

Looking at SAR this morning, I can see that I'm sometime reaching
300tps, and spiked at 80% WIO... That will cost me 5 new additional
hard drives :(

I’m not sure I will have time to install them all today, but as soon
as it’s done I will give you some news.

JM
2013/2/8, Kevin O'dell <[EMAIL PROTECTED]>:
> JM,
>
>   Basically, you will have to replace failed disk and rebuild RAID0 since
> the other half of the data is worthless.  There is not a real recommended
> value, but anything under 150 - 200 would make me more comfortable.
>
> On Fri, Feb 8, 2013 at 10:43 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Kevin,
>>
>> I think it will take time before I get a chance to have 5 drives in
>> the same server, so I will see at that time to test RAID5.
>>
>> I'm going to add one drive per server today or tomorrow to try to
>> improve that. What IOPs should I try to have? 100? Less? It will all
>> be SATA3 drives and I will configure all in RAID0.
>>
>> It doesn't seems to me to be an issue to lose one node, since data
>> will be replicated everywhere else. I will "simply" have to replace
>> the failing disk and restart the node, no?
>>
>> JM
>>
>> 2013/2/8, Kevin O'dell <[EMAIL PROTECTED]>:
>> > Azuryy,
>> >
>> >   The main reason to recommend against RAID is that it is slow and it
>> adds
>> > redundancy that we already have in Hadoop.  RAID0 is another story as
>> long
>> > as all of the drives are healthy and you don't mind losing the whole
>> volume
>> > if you lose one drive.
>> >
>> > JM,
>> >
>> >   I would not even waste my time testing RAID5 or RAID6(unless it is
>> > just
>> > for educational purposes :) ).  200+ IOPs consistently on one SATA
>> > drive
>> is
>> > pretty high, that would explain your high I/O wait time.  If your use
>> case
>> > allows for you to lose the whole node, there is not a good reason for
>> > you
>> > to shy away from RAID0.  Please let us know how this plays out with
>> > your
>> > environment.
>> >
>> > On Thu, Feb 7, 2013 at 10:23 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>> >
>> >> JM，
>> >>
>> >> I don't have the context, but if you are using Hadoop/Hbase, so don't
>> >> do
>> >> RAID on your disk.
>> >>
>> >>
>> >> On Fri, Feb 8, 2013 at 11:15 AM, Jean-Marc Spaggiari <
>> >> [EMAIL PROTECTED]> wrote:
>> >>
>> >> > Ok. I see. For my usecase I prefer to loose the data and have faster
>> >> > process. So I will go for RAID0 and keep the replication factor to
>> >> > 3... If at some point I have 5 disks in the node, I will most
>> >> > probably
>> >> > give a try to RAID5 and see the performances compared to the other
>> >> > RAID/JBOD options.
>> >> >
>> >> > Is there a "rule", like, 1 HD per core? Or we can't really simplify
>> >> > that
>> >> > much?
>> >> >
>> >> > So far I have that in the sar output:
>> >> > 21:35:03          tps      rtps      wtps   bread/s   bwrtn/s
>> >> > 21:45:03       218,85    215,97      2,88  45441,95    308,04
>> >> > 21:55:02       209,73    206,67      3,06  43985,28    378,32
>> >> > 22:05:04       215,03    211,71      3,33  44831,00    312,95
>> >> > Average :      214,54    211,45      3,09  44753,09    333,07
>> >> >
>> >> > But I'm not sure what it means. I will wait for tomorrow to get more
>> >> > results, but my job will be done over night, so I'm not sure the
>> >> > average will be accurate...
>> >> >
>> >> > JM
>> >> >
>> >> >
>> >> > 2013/2/7, Kevin O'dell <[EMAIL PROTECTED]>:
>> >> > > JM,
>> >> > >
>> >> > >   I think you misunderstood me.  I am not advocating any form of
>> RAID
>> >> for
>> >> > > Hadoop.  It is true that we already have redundancy built in with
>> >> > > HDFS.
>> >> >  So
>> >> > > unless you were going to do something silly like sacrifice speed
>> >> > > to
>> >> > > run
>> >> > > RAID1 or RAID5 and lower your replication to 2...just don't do it
>> >> > > :)
>> >> > >  Anyway, yes you probably should have 3 - 4 drives per node if not
>> >> more.
>> >> > >  At that point then the you will really see the benefit of JBOD