Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> recommended nodes


+
David Charle 2012-11-26, 13:53
+
Marcos Ortiz 2012-11-26, 14:05
+
Michael Segel 2012-11-26, 14:43
+
Mohammad Tariq 2012-11-26, 13:59
+
Jean-Marc Spaggiari 2012-12-19, 22:59
+
Michael Segel 2012-12-20, 01:14
Copy link to this message
-
Re: recommended nodes
Hi Jean,

Very interesting benchmark - how are these numbers arrived at ? Is this on
a real hbase cluster ? To me, it felt kind of counter intuitive that RAID0
beats JBOD on random seeks because with RAID0 all disks need to seek at the
same time and the performance should basically be as bad as the slowest
seeking disk.

Varun

On Wed, Dec 19, 2012 at 5:14 PM, Michael Segel <[EMAIL PROTECTED]>wrote:

> Yeah,
> I couldn't argue against LVMs when talking with the system admins.
> In terms of speed its noise because the CPUs are pretty efficient and
> unless you have more than 1 drive per physical core, you will end up
> saturating your disk I/O.
>
> In terms of MapR, you want the raw disk. (But we're talking Apache)
>
>
> On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]>
> wrote:
>
> > Finally, it took me a while to run those tests because it was way
> > longer than expected, but here are the results:
> >
> > http://www.spaggiari.org/bonnie.html
> >
> > LVM is not really slower than JBOD and not really taking more CPU. So
> > I will say, if you have to choose between the 2, take the one you
> > prefer. Personally, I prefer LVM because it's easy to configure.
> >
> > The big winner here is RAID0. It's WAY faster than anything else. But
> > it's using twice the space... Your choice.
> >
> > I did not get a chance to test with the Ubuntu tool because it's not
> > working with LVM drives.
> >
> > JM
> >
> > 2012/11/28, Michael Segel <[EMAIL PROTECTED]>:
> >> Ok, just a caveat.
> >>
> >> I am discussing MapR as part of a complete response. As Mohit posted
> MapR
> >> takes the raw device for their MapR File System.
> >> They do stripe on their own within what they call a volume.
> >>
> >> But going back to Apache...
> >> You can stripe drives, however I wouldn't recommend it. I don't think
> the
> >> performance gains would really matter.
> >> You're going to end up getting blocked first by disk i/o, then your
> >> controller card, then your network... assuming 10GBe.
> >>
> >> With only 2 disks on an 8 core system, you will hit disk i/o first and
> then
> >> you'll watch your CPU Wait I/O climb.
> >>
> >> HTH
> >>
> >> -Mike
> >>
> >> On Nov 28, 2012, at 7:28 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]>
> >> wrote:
> >>
> >>> Hi Mike,
> >>>
> >>> Why not using LVM with MapR? Since LVM is reading from 2 drives almost
> >>> at the same time, it should be better than RAID0 or a single drive,
> >>> no?
> >>>
> >>> 2012/11/28, Michael Segel <[EMAIL PROTECTED]>:
> >>>> Just a couple of things.
> >>>>
> >>>> I'm neutral on the use of LVMs. Some would point out that there's some
> >>>> overhead, but on the flip side, it can make managing the machines
> >>>> easier.
> >>>> If you're using MapR, you don't want to use LVMs but raw devices.
> >>>>
> >>>> In terms of GC, its going to depend on the heap size and not the total
> >>>> memory. With respect to HBase. ... MSLABS is the way to go.
> >>>>
> >>>>
> >>>> On Nov 28, 2012, at 12:05 PM, Jean-Marc Spaggiari
> >>>> <[EMAIL PROTECTED]>
> >>>> wrote:
> >>>>
> >>>>> Hi Gregory,
> >>>>>
> >>>>> I founs this about LVM:
> >>>>> -> http://blog.andrew.net.au/2006/08/09
> >>>>> ->
> >>>>>
> http://www.phoronix.com/scan.php?page=article&item=fedora_15_lvm&num=2
> >>>>>
> >>>>> Seems that performances are still correct with it. I will most
> >>>>> probably give it a try and bench that too... I have one new hard
> drive
> >>>>> which should arrived tomorrow. Perfect timing ;)
> >>>>>
> >>>>>
> >>>>>
> >>>>> JM
> >>>>>
> >>>>> 2012/11/28, Mohit Anchlia <[EMAIL PROTECTED]>:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Nov 28, 2012, at 9:07 AM, Adrien Mogenet <
> [EMAIL PROTECTED]>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Does HBase really benefit from 64 GB of RAM since allocating too
> >>>>>>> large
> >>>>>>> heap
> >>>>>>> might increase GC time ?
> >>>>>>>
> >>>>>> Benefit you get is from OS cache
> >>>>>>> Another question : why not RAID 0, in order to aggregate disk
+
Jean-Marc Spaggiari 2012-12-20, 21:26
+
Varun Sharma 2012-12-20, 21:37
+
Jean-Marc Spaggiari 2012-12-20, 22:07
+
Adrien Mogenet 2012-12-20, 22:11