|
|
David Charle 2012-11-26, 13:53
hi
what's the recommended nodes for NN, hmaster and zk nodes for a larger cluster, lets say 50-100+
also, what would be the ideal replication factor for larger clusters when u have 3-4 racks ?
--
David
+
David Charle 2012-11-26, 13:53
+
Marcos Ortiz 2012-11-26, 14:05
Michael Segel 2012-11-26, 14:43
Uhm, those specs are actually now out of date. If you're running HBase, or want to also run R on top of Hadoop, you will need to add more memory. Also forget 1GBe got 10GBe, and w 2 SATA drives, you will be disk i/o bound way too quickly. On Nov 26, 2012, at 8:05 AM, Marcos Ortiz <[EMAIL PROTECTED]> wrote: > Are you asking about hardware recommendations? > Eric Sammer on his "Hadoop Operations" book, did a great job about this: > For middle size clusters (until 300 nodes): > Processor: A dual quad-core 2.6 Ghz > RAM: 24 GB DDR3 > Dual 1 Gb Ethernet NICs > a SAS drive controller > at least two SATA II drives in a JBOD configuration > > The replication factor depends heavily of the primary use of your cluster. > > On 11/26/2012 08:53 AM, David Charle wrote: >> hi >> >> what's the recommended nodes for NN, hmaster and zk nodes for a larger cluster, lets say 50-100+ >> >> also, what would be the ideal replication factor for larger clusters when u have 3-4 racks ? >> >> -- >> David >> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS... >> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION >> >> http://www.uci.cu>> http://www.facebook.com/universidad.uci>> http://www.flickr.com/photos/universidad_uci> > -- > > Marcos Luis Ortíz Valmaseda > about.me/marcosortiz < http://about.me/marcosortiz>> @marcosluis2186 < http://twitter.com/marcosluis2186>> > > > 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS... > CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION > > http://www.uci.cu> http://www.facebook.com/universidad.uci> http://www.flickr.com/photos/universidad_uci
+
Michael Segel 2012-11-26, 14:43
Mohammad Tariq 2012-11-26, 13:59
Hello David,
Do you mean the recommended specs?IMHO, it depends more on the data
and the kind of processing you are going to perform, rather than the size
of your cluster.
Regards,
Mohammad Tariq
On Mon, Nov 26, 2012 at 7:23 PM, David Charle <[EMAIL PROTECTED]> wrote:
> hi
>
> what's the recommended nodes for NN, hmaster and zk nodes for a larger
> cluster, lets say 50-100+
>
> also, what would be the ideal replication factor for larger clusters when
> u have 3-4 racks ?
>
> --
> David
+
Mohammad Tariq 2012-11-26, 13:59
Jean-Marc Spaggiari 2012-12-19, 22:59
Finally, it took me a while to run those tests because it was way longer than expected, but here are the results: http://www.spaggiari.org/bonnie.htmlLVM is not really slower than JBOD and not really taking more CPU. So I will say, if you have to choose between the 2, take the one you prefer. Personally, I prefer LVM because it's easy to configure. The big winner here is RAID0. It's WAY faster than anything else. But it's using twice the space... Your choice. I did not get a chance to test with the Ubuntu tool because it's not working with LVM drives. JM 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: > Ok, just a caveat. > > I am discussing MapR as part of a complete response. As Mohit posted MapR > takes the raw device for their MapR File System. > They do stripe on their own within what they call a volume. > > But going back to Apache... > You can stripe drives, however I wouldn't recommend it. I don't think the > performance gains would really matter. > You're going to end up getting blocked first by disk i/o, then your > controller card, then your network... assuming 10GBe. > > With only 2 disks on an 8 core system, you will hit disk i/o first and then > you'll watch your CPU Wait I/O climb. > > HTH > > -Mike > > On Nov 28, 2012, at 7:28 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]> > wrote: > >> Hi Mike, >> >> Why not using LVM with MapR? Since LVM is reading from 2 drives almost >> at the same time, it should be better than RAID0 or a single drive, >> no? >> >> 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: >>> Just a couple of things. >>> >>> I'm neutral on the use of LVMs. Some would point out that there's some >>> overhead, but on the flip side, it can make managing the machines >>> easier. >>> If you're using MapR, you don't want to use LVMs but raw devices. >>> >>> In terms of GC, its going to depend on the heap size and not the total >>> memory. With respect to HBase. ... MSLABS is the way to go. >>> >>> >>> On Nov 28, 2012, at 12:05 PM, Jean-Marc Spaggiari >>> <[EMAIL PROTECTED]> >>> wrote: >>> >>>> Hi Gregory, >>>> >>>> I founs this about LVM: >>>> -> http://blog.andrew.net.au/2006/08/09>>>> -> >>>> http://www.phoronix.com/scan.php?page=article&item=fedora_15_lvm&num=2>>>> >>>> Seems that performances are still correct with it. I will most >>>> probably give it a try and bench that too... I have one new hard drive >>>> which should arrived tomorrow. Perfect timing ;) >>>> >>>> >>>> >>>> JM >>>> >>>> 2012/11/28, Mohit Anchlia <[EMAIL PROTECTED]>: >>>>> >>>>> >>>>> >>>>> >>>>> On Nov 28, 2012, at 9:07 AM, Adrien Mogenet <[EMAIL PROTECTED]> >>>>> wrote: >>>>> >>>>>> Does HBase really benefit from 64 GB of RAM since allocating too >>>>>> large >>>>>> heap >>>>>> might increase GC time ? >>>>>> >>>>> Benefit you get is from OS cache >>>>>> Another question : why not RAID 0, in order to aggregate disk >>>>>> bandwidth >>>>>> ? >>>>>> (and thus keep 3x replication factor) >>>>>> >>>>>> >>>>>> On Wed, Nov 28, 2012 at 5:58 PM, Michael Segel >>>>>> <[EMAIL PROTECTED]>wrote: >>>>>> >>>>>>> Sorry, >>>>>>> >>>>>>> I need to clarify. >>>>>>> >>>>>>> 4GB per physical core is a good starting point. >>>>>>> So with 2 quad core chips, that is going to be 32GB. >>>>>>> >>>>>>> IMHO that's a minimum. If you go with HBase, you will want more. >>>>>>> (Actually >>>>>>> you will need more.) The next logical jump would be to 48 or 64GB. >>>>>>> >>>>>>> If we start to price out memory, depending on vendor, your company's >>>>>>> procurement, there really isn't much of a price difference in terms >>>>>>> of >>>>>>> 32,48, or 64 GB. >>>>>>> Note that it also depends on the chips themselves. Also you need to >>>>>>> see >>>>>>> how many memory channels exist in the mother board. You may need to >>>>>>> buy >>>>>>> in >>>>>>> pairs or triplets. Your hardware vendor can help you. (Also you need >>>>>>> to >>>>>>> keep an eye on your hardware vendor. Sometimes they will give you >>>>>>> higher
+
Jean-Marc Spaggiari 2012-12-19, 22:59
Michael Segel 2012-12-20, 01:14
Yeah, I couldn't argue against LVMs when talking with the system admins. In terms of speed its noise because the CPUs are pretty efficient and unless you have more than 1 drive per physical core, you will end up saturating your disk I/O. In terms of MapR, you want the raw disk. (But we're talking Apache) On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]> wrote: > Finally, it took me a while to run those tests because it was way > longer than expected, but here are the results: > > http://www.spaggiari.org/bonnie.html> > LVM is not really slower than JBOD and not really taking more CPU. So > I will say, if you have to choose between the 2, take the one you > prefer. Personally, I prefer LVM because it's easy to configure. > > The big winner here is RAID0. It's WAY faster than anything else. But > it's using twice the space... Your choice. > > I did not get a chance to test with the Ubuntu tool because it's not > working with LVM drives. > > JM > > 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: >> Ok, just a caveat. >> >> I am discussing MapR as part of a complete response. As Mohit posted MapR >> takes the raw device for their MapR File System. >> They do stripe on their own within what they call a volume. >> >> But going back to Apache... >> You can stripe drives, however I wouldn't recommend it. I don't think the >> performance gains would really matter. >> You're going to end up getting blocked first by disk i/o, then your >> controller card, then your network... assuming 10GBe. >> >> With only 2 disks on an 8 core system, you will hit disk i/o first and then >> you'll watch your CPU Wait I/O climb. >> >> HTH >> >> -Mike >> >> On Nov 28, 2012, at 7:28 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]> >> wrote: >> >>> Hi Mike, >>> >>> Why not using LVM with MapR? Since LVM is reading from 2 drives almost >>> at the same time, it should be better than RAID0 or a single drive, >>> no? >>> >>> 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: >>>> Just a couple of things. >>>> >>>> I'm neutral on the use of LVMs. Some would point out that there's some >>>> overhead, but on the flip side, it can make managing the machines >>>> easier. >>>> If you're using MapR, you don't want to use LVMs but raw devices. >>>> >>>> In terms of GC, its going to depend on the heap size and not the total >>>> memory. With respect to HBase. ... MSLABS is the way to go. >>>> >>>> >>>> On Nov 28, 2012, at 12:05 PM, Jean-Marc Spaggiari >>>> <[EMAIL PROTECTED]> >>>> wrote: >>>> >>>>> Hi Gregory, >>>>> >>>>> I founs this about LVM: >>>>> -> http://blog.andrew.net.au/2006/08/09>>>>> -> >>>>> http://www.phoronix.com/scan.php?page=article&item=fedora_15_lvm&num=2>>>>> >>>>> Seems that performances are still correct with it. I will most >>>>> probably give it a try and bench that too... I have one new hard drive >>>>> which should arrived tomorrow. Perfect timing ;) >>>>> >>>>> >>>>> >>>>> JM >>>>> >>>>> 2012/11/28, Mohit Anchlia <[EMAIL PROTECTED]>: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Nov 28, 2012, at 9:07 AM, Adrien Mogenet <[EMAIL PROTECTED]> >>>>>> wrote: >>>>>> >>>>>>> Does HBase really benefit from 64 GB of RAM since allocating too >>>>>>> large >>>>>>> heap >>>>>>> might increase GC time ? >>>>>>> >>>>>> Benefit you get is from OS cache >>>>>>> Another question : why not RAID 0, in order to aggregate disk >>>>>>> bandwidth >>>>>>> ? >>>>>>> (and thus keep 3x replication factor) >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 28, 2012 at 5:58 PM, Michael Segel >>>>>>> <[EMAIL PROTECTED]>wrote: >>>>>>> >>>>>>>> Sorry, >>>>>>>> >>>>>>>> I need to clarify. >>>>>>>> >>>>>>>> 4GB per physical core is a good starting point. >>>>>>>> So with 2 quad core chips, that is going to be 32GB. >>>>>>>> >>>>>>>> IMHO that's a minimum. If you go with HBase, you will want more. >>>>>>>> (Actually >>>>>>>> you will need more.) The next logical jump would be to 48 or 64GB. >>
+
Michael Segel 2012-12-20, 01:14
Varun Sharma 2012-12-20, 21:22
Hi Jean, Very interesting benchmark - how are these numbers arrived at ? Is this on a real hbase cluster ? To me, it felt kind of counter intuitive that RAID0 beats JBOD on random seeks because with RAID0 all disks need to seek at the same time and the performance should basically be as bad as the slowest seeking disk. Varun On Wed, Dec 19, 2012 at 5:14 PM, Michael Segel <[EMAIL PROTECTED]>wrote: > Yeah, > I couldn't argue against LVMs when talking with the system admins. > In terms of speed its noise because the CPUs are pretty efficient and > unless you have more than 1 drive per physical core, you will end up > saturating your disk I/O. > > In terms of MapR, you want the raw disk. (But we're talking Apache) > > > On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]> > wrote: > > > Finally, it took me a while to run those tests because it was way > > longer than expected, but here are the results: > > > > http://www.spaggiari.org/bonnie.html> > > > LVM is not really slower than JBOD and not really taking more CPU. So > > I will say, if you have to choose between the 2, take the one you > > prefer. Personally, I prefer LVM because it's easy to configure. > > > > The big winner here is RAID0. It's WAY faster than anything else. But > > it's using twice the space... Your choice. > > > > I did not get a chance to test with the Ubuntu tool because it's not > > working with LVM drives. > > > > JM > > > > 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: > >> Ok, just a caveat. > >> > >> I am discussing MapR as part of a complete response. As Mohit posted > MapR > >> takes the raw device for their MapR File System. > >> They do stripe on their own within what they call a volume. > >> > >> But going back to Apache... > >> You can stripe drives, however I wouldn't recommend it. I don't think > the > >> performance gains would really matter. > >> You're going to end up getting blocked first by disk i/o, then your > >> controller card, then your network... assuming 10GBe. > >> > >> With only 2 disks on an 8 core system, you will hit disk i/o first and > then > >> you'll watch your CPU Wait I/O climb. > >> > >> HTH > >> > >> -Mike > >> > >> On Nov 28, 2012, at 7:28 PM, Jean-Marc Spaggiari < > [EMAIL PROTECTED]> > >> wrote: > >> > >>> Hi Mike, > >>> > >>> Why not using LVM with MapR? Since LVM is reading from 2 drives almost > >>> at the same time, it should be better than RAID0 or a single drive, > >>> no? > >>> > >>> 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: > >>>> Just a couple of things. > >>>> > >>>> I'm neutral on the use of LVMs. Some would point out that there's some > >>>> overhead, but on the flip side, it can make managing the machines > >>>> easier. > >>>> If you're using MapR, you don't want to use LVMs but raw devices. > >>>> > >>>> In terms of GC, its going to depend on the heap size and not the total > >>>> memory. With respect to HBase. ... MSLABS is the way to go. > >>>> > >>>> > >>>> On Nov 28, 2012, at 12:05 PM, Jean-Marc Spaggiari > >>>> <[EMAIL PROTECTED]> > >>>> wrote: > >>>> > >>>>> Hi Gregory, > >>>>> > >>>>> I founs this about LVM: > >>>>> -> http://blog.andrew.net.au/2006/08/09> >>>>> -> > >>>>> > http://www.phoronix.com/scan.php?page=article&item=fedora_15_lvm&num=2> >>>>> > >>>>> Seems that performances are still correct with it. I will most > >>>>> probably give it a try and bench that too... I have one new hard > drive > >>>>> which should arrived tomorrow. Perfect timing ;) > >>>>> > >>>>> > >>>>> > >>>>> JM > >>>>> > >>>>> 2012/11/28, Mohit Anchlia <[EMAIL PROTECTED]>: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Nov 28, 2012, at 9:07 AM, Adrien Mogenet < > [EMAIL PROTECTED]> > >>>>>> wrote: > >>>>>> > >>>>>>> Does HBase really benefit from 64 GB of RAM since allocating too > >>>>>>> large > >>>>>>> heap > >>>>>>> might increase GC time ? > >>>>>>> > >>>>>> Benefit you get is from OS cache > >>>>>>> Another question : why not RAID 0, in order to aggregate disk
+
Varun Sharma 2012-12-20, 21:22
Jean-Marc Spaggiari 2012-12-20, 21:26
Hi Varun, The hard drivers I used are now used on the hadoop/hbase cluster, but they was clear and formated for the tests I did. The computer where I run those tests was one of the region servers. It was re-installed to be very clear, and it's now running a datanode and a RS. Regarding RAID, I think you are confusing RAID0 and RAID1. It's RAID1 which need to access the 2 files each time. RAID0 is more like JBOD, but faster. JM 2012/12/20 Varun Sharma <[EMAIL PROTECTED]> > Hi Jean, > > Very interesting benchmark - how are these numbers arrived at ? Is this on > a real hbase cluster ? To me, it felt kind of counter intuitive that RAID0 > beats JBOD on random seeks because with RAID0 all disks need to seek at the > same time and the performance should basically be as bad as the slowest > seeking disk. > > Varun > > On Wed, Dec 19, 2012 at 5:14 PM, Michael Segel <[EMAIL PROTECTED] > >wrote: > > > Yeah, > > I couldn't argue against LVMs when talking with the system admins. > > In terms of speed its noise because the CPUs are pretty efficient and > > unless you have more than 1 drive per physical core, you will end up > > saturating your disk I/O. > > > > In terms of MapR, you want the raw disk. (But we're talking Apache) > > > > > > On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari < > [EMAIL PROTECTED]> > > wrote: > > > > > Finally, it took me a while to run those tests because it was way > > > longer than expected, but here are the results: > > > > > > http://www.spaggiari.org/bonnie.html> > > > > > LVM is not really slower than JBOD and not really taking more CPU. So > > > I will say, if you have to choose between the 2, take the one you > > > prefer. Personally, I prefer LVM because it's easy to configure. > > > > > > The big winner here is RAID0. It's WAY faster than anything else. But > > > it's using twice the space... Your choice. > > > > > > I did not get a chance to test with the Ubuntu tool because it's not > > > working with LVM drives. > > > > > > JM > > > > > > 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: > > >> Ok, just a caveat. > > >> > > >> I am discussing MapR as part of a complete response. As Mohit posted > > MapR > > >> takes the raw device for their MapR File System. > > >> They do stripe on their own within what they call a volume. > > >> > > >> But going back to Apache... > > >> You can stripe drives, however I wouldn't recommend it. I don't think > > the > > >> performance gains would really matter. > > >> You're going to end up getting blocked first by disk i/o, then your > > >> controller card, then your network... assuming 10GBe. > > >> > > >> With only 2 disks on an 8 core system, you will hit disk i/o first and > > then > > >> you'll watch your CPU Wait I/O climb. > > >> > > >> HTH > > >> > > >> -Mike > > >> > > >> On Nov 28, 2012, at 7:28 PM, Jean-Marc Spaggiari < > > [EMAIL PROTECTED]> > > >> wrote: > > >> > > >>> Hi Mike, > > >>> > > >>> Why not using LVM with MapR? Since LVM is reading from 2 drives > almost > > >>> at the same time, it should be better than RAID0 or a single drive, > > >>> no? > > >>> > > >>> 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: > > >>>> Just a couple of things. > > >>>> > > >>>> I'm neutral on the use of LVMs. Some would point out that there's > some > > >>>> overhead, but on the flip side, it can make managing the machines > > >>>> easier. > > >>>> If you're using MapR, you don't want to use LVMs but raw devices. > > >>>> > > >>>> In terms of GC, its going to depend on the heap size and not the > total > > >>>> memory. With respect to HBase. ... MSLABS is the way to go. > > >>>> > > >>>> > > >>>> On Nov 28, 2012, at 12:05 PM, Jean-Marc Spaggiari > > >>>> <[EMAIL PROTECTED]> > > >>>> wrote: > > >>>> > > >>>>> Hi Gregory, > > >>>>> > > >>>>> I founs this about LVM: > > >>>>> -> http://blog.andrew.net.au/2006/08/09> > >>>>> -> > > >>>>> > > http://www.phoronix.com/scan.php?page=article&item=fedora_15_lvm&num=2> > >>>>>
+
Jean-Marc Spaggiari 2012-12-20, 21:26
Varun Sharma 2012-12-20, 21:37
Hmm, I thought that RAID0 simply stripes across all disks. So if you got 4 disks - an HFile block for example could get striped across 4 disks. So to read that block, you would need all 4 of them to seek so that you could read all 4 stripes for that HFile block. This could make things as slow as the slowest seeking disk for that random read. However, certainly, data xfer rate would be much faster with RAID0 but since this is merely 64K for a HFile block, I would have expected the seek latency to play a major role and not really the data xfer latency. However, your tests indeed show that RAID0 still outperforms JBOD on seeks. Am I missing something ? On Thu, Dec 20, 2012 at 1:26 PM, Jean-Marc Spaggiari < [EMAIL PROTECTED]> wrote: > Hi Varun, > > The hard drivers I used are now used on the hadoop/hbase cluster, but they > was clear and formated for the tests I did. The computer where I run those > tests was one of the region servers. It was re-installed to be very clear, > and it's now running a datanode and a RS. > > Regarding RAID, I think you are confusing RAID0 and RAID1. It's RAID1 which > need to access the 2 files each time. RAID0 is more like JBOD, but faster. > > JM > > 2012/12/20 Varun Sharma <[EMAIL PROTECTED]> > > > Hi Jean, > > > > Very interesting benchmark - how are these numbers arrived at ? Is this > on > > a real hbase cluster ? To me, it felt kind of counter intuitive that > RAID0 > > beats JBOD on random seeks because with RAID0 all disks need to seek at > the > > same time and the performance should basically be as bad as the slowest > > seeking disk. > > > > Varun > > > > On Wed, Dec 19, 2012 at 5:14 PM, Michael Segel < > [EMAIL PROTECTED] > > >wrote: > > > > > Yeah, > > > I couldn't argue against LVMs when talking with the system admins. > > > In terms of speed its noise because the CPUs are pretty efficient and > > > unless you have more than 1 drive per physical core, you will end up > > > saturating your disk I/O. > > > > > > In terms of MapR, you want the raw disk. (But we're talking Apache) > > > > > > > > > On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari < > > [EMAIL PROTECTED]> > > > wrote: > > > > > > > Finally, it took me a while to run those tests because it was way > > > > longer than expected, but here are the results: > > > > > > > > http://www.spaggiari.org/bonnie.html> > > > > > > > LVM is not really slower than JBOD and not really taking more CPU. So > > > > I will say, if you have to choose between the 2, take the one you > > > > prefer. Personally, I prefer LVM because it's easy to configure. > > > > > > > > The big winner here is RAID0. It's WAY faster than anything else. But > > > > it's using twice the space... Your choice. > > > > > > > > I did not get a chance to test with the Ubuntu tool because it's not > > > > working with LVM drives. > > > > > > > > JM > > > > > > > > 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: > > > >> Ok, just a caveat. > > > >> > > > >> I am discussing MapR as part of a complete response. As Mohit posted > > > MapR > > > >> takes the raw device for their MapR File System. > > > >> They do stripe on their own within what they call a volume. > > > >> > > > >> But going back to Apache... > > > >> You can stripe drives, however I wouldn't recommend it. I don't > think > > > the > > > >> performance gains would really matter. > > > >> You're going to end up getting blocked first by disk i/o, then your > > > >> controller card, then your network... assuming 10GBe. > > > >> > > > >> With only 2 disks on an 8 core system, you will hit disk i/o first > and > > > then > > > >> you'll watch your CPU Wait I/O climb. > > > >> > > > >> HTH > > > >> > > > >> -Mike > > > >> > > > >> On Nov 28, 2012, at 7:28 PM, Jean-Marc Spaggiari < > > > [EMAIL PROTECTED]> > > > >> wrote: > > > >> > > > >>> Hi Mike, > > > >>> > > > >>> Why not using LVM with MapR? Since LVM is reading from 2 drives > > almost > > > >>> at the same time, it should be better than RAID0 or a single drive,
+
Varun Sharma 2012-12-20, 21:37
Jean-Marc Spaggiari 2012-12-20, 22:07
I did the test with a 2GB file... So read and write were spread over the 2 drives for RAID0. Those test were to give an overall idea of the performances vs CPU usage etc. and you might need to adjust them based on the way it's configured on your system. I don't know how RAID0 is managing small files (<=64k) but maybe it's still spread on the 2 disks too? JM 2012/12/20 Varun Sharma <[EMAIL PROTECTED]> > Hmm, I thought that RAID0 simply stripes across all disks. So if you got 4 > disks - an HFile block for example could get striped across 4 disks. So to > read that block, you would need all 4 of them to seek so that you could > read all 4 stripes for that HFile block. This could make things as slow as > the slowest seeking disk for that random read. However, certainly, data > xfer rate would be much faster with RAID0 but since this is merely 64K for > a HFile block, I would have expected the seek latency to play a major role > and not really the data xfer latency. > > However, your tests indeed show that RAID0 still outperforms JBOD on seeks. > Am I missing something ? > > On Thu, Dec 20, 2012 at 1:26 PM, Jean-Marc Spaggiari < > [EMAIL PROTECTED]> wrote: > > > Hi Varun, > > > > The hard drivers I used are now used on the hadoop/hbase cluster, but > they > > was clear and formated for the tests I did. The computer where I run > those > > tests was one of the region servers. It was re-installed to be very > clear, > > and it's now running a datanode and a RS. > > > > Regarding RAID, I think you are confusing RAID0 and RAID1. It's RAID1 > which > > need to access the 2 files each time. RAID0 is more like JBOD, but > faster. > > > > JM > > > > 2012/12/20 Varun Sharma <[EMAIL PROTECTED]> > > > > > Hi Jean, > > > > > > Very interesting benchmark - how are these numbers arrived at ? Is this > > on > > > a real hbase cluster ? To me, it felt kind of counter intuitive that > > RAID0 > > > beats JBOD on random seeks because with RAID0 all disks need to seek at > > the > > > same time and the performance should basically be as bad as the slowest > > > seeking disk. > > > > > > Varun > > > > > > On Wed, Dec 19, 2012 at 5:14 PM, Michael Segel < > > [EMAIL PROTECTED] > > > >wrote: > > > > > > > Yeah, > > > > I couldn't argue against LVMs when talking with the system admins. > > > > In terms of speed its noise because the CPUs are pretty efficient and > > > > unless you have more than 1 drive per physical core, you will end up > > > > saturating your disk I/O. > > > > > > > > In terms of MapR, you want the raw disk. (But we're talking Apache) > > > > > > > > > > > > On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari < > > > [EMAIL PROTECTED]> > > > > wrote: > > > > > > > > > Finally, it took me a while to run those tests because it was way > > > > > longer than expected, but here are the results: > > > > > > > > > > http://www.spaggiari.org/bonnie.html> > > > > > > > > > LVM is not really slower than JBOD and not really taking more CPU. > So > > > > > I will say, if you have to choose between the 2, take the one you > > > > > prefer. Personally, I prefer LVM because it's easy to configure. > > > > > > > > > > The big winner here is RAID0. It's WAY faster than anything else. > But > > > > > it's using twice the space... Your choice. > > > > > > > > > > I did not get a chance to test with the Ubuntu tool because it's > not > > > > > working with LVM drives. > > > > > > > > > > JM > > > > > > > > > > 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: > > > > >> Ok, just a caveat. > > > > >> > > > > >> I am discussing MapR as part of a complete response. As Mohit > posted > > > > MapR > > > > >> takes the raw device for their MapR File System. > > > > >> They do stripe on their own within what they call a volume. > > > > >> > > > > >> But going back to Apache... > > > > >> You can stripe drives, however I wouldn't recommend it. I don't > > think > > > > the > > > > >> performance gains would really matter. > > > > >
+
Jean-Marc Spaggiari 2012-12-20, 22:07
Adrien Mogenet 2012-12-20, 22:11
Maybe you should give a little more information about your RAID controller (write back / write through ?) and the underlying filesystem (ext3 ? blocksize ?). Very interesting benchmark and discussion by the way :-) On Thu, Dec 20, 2012 at 11:07 PM, Jean-Marc Spaggiari < [EMAIL PROTECTED]> wrote: > I did the test with a 2GB file... So read and write were spread over the 2 > drives for RAID0. > > Those test were to give an overall idea of the performances vs CPU usage > etc. and you might need to adjust them based on the way it's configured on > your system. > > I don't know how RAID0 is managing small files (<=64k) but maybe it's still > spread on the 2 disks too? > > JM > > 2012/12/20 Varun Sharma <[EMAIL PROTECTED]> > > > Hmm, I thought that RAID0 simply stripes across all disks. So if you got > 4 > > disks - an HFile block for example could get striped across 4 disks. So > to > > read that block, you would need all 4 of them to seek so that you could > > read all 4 stripes for that HFile block. This could make things as slow > as > > the slowest seeking disk for that random read. However, certainly, data > > xfer rate would be much faster with RAID0 but since this is merely 64K > for > > a HFile block, I would have expected the seek latency to play a major > role > > and not really the data xfer latency. > > > > However, your tests indeed show that RAID0 still outperforms JBOD on > seeks. > > Am I missing something ? > > > > On Thu, Dec 20, 2012 at 1:26 PM, Jean-Marc Spaggiari < > > [EMAIL PROTECTED]> wrote: > > > > > Hi Varun, > > > > > > The hard drivers I used are now used on the hadoop/hbase cluster, but > > they > > > was clear and formated for the tests I did. The computer where I run > > those > > > tests was one of the region servers. It was re-installed to be very > > clear, > > > and it's now running a datanode and a RS. > > > > > > Regarding RAID, I think you are confusing RAID0 and RAID1. It's RAID1 > > which > > > need to access the 2 files each time. RAID0 is more like JBOD, but > > faster. > > > > > > JM > > > > > > 2012/12/20 Varun Sharma <[EMAIL PROTECTED]> > > > > > > > Hi Jean, > > > > > > > > Very interesting benchmark - how are these numbers arrived at ? Is > this > > > on > > > > a real hbase cluster ? To me, it felt kind of counter intuitive that > > > RAID0 > > > > beats JBOD on random seeks because with RAID0 all disks need to seek > at > > > the > > > > same time and the performance should basically be as bad as the > slowest > > > > seeking disk. > > > > > > > > Varun > > > > > > > > On Wed, Dec 19, 2012 at 5:14 PM, Michael Segel < > > > [EMAIL PROTECTED] > > > > >wrote: > > > > > > > > > Yeah, > > > > > I couldn't argue against LVMs when talking with the system admins. > > > > > In terms of speed its noise because the CPUs are pretty efficient > and > > > > > unless you have more than 1 drive per physical core, you will end > up > > > > > saturating your disk I/O. > > > > > > > > > > In terms of MapR, you want the raw disk. (But we're talking Apache) > > > > > > > > > > > > > > > On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari < > > > > [EMAIL PROTECTED]> > > > > > wrote: > > > > > > > > > > > Finally, it took me a while to run those tests because it was way > > > > > > longer than expected, but here are the results: > > > > > > > > > > > > http://www.spaggiari.org/bonnie.html> > > > > > > > > > > > LVM is not really slower than JBOD and not really taking more > CPU. > > So > > > > > > I will say, if you have to choose between the 2, take the one you > > > > > > prefer. Personally, I prefer LVM because it's easy to configure. > > > > > > > > > > > > The big winner here is RAID0. It's WAY faster than anything else. > > But > > > > > > it's using twice the space... Your choice. > > > > > > > > > > > > I did not get a chance to test with the Ubuntu tool because it's > > not > > > > > > working with LVM drives. > > > > > > > > > > > > JM > > > > > > > > > > > > 2012/11/28, Michael Segel <[EMAIL PROTECTED]>: Adrien Mogenet 06.59.16.64.22 http://www.mogenet.me
+
Adrien Mogenet 2012-12-20, 22:11
|
|
