Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> recommended nodes


+
David Charle 2012-11-26, 13:53
+
Marcos Ortiz 2012-11-26, 14:05
+
Michael Segel 2012-11-26, 14:43
+
Mohammad Tariq 2012-11-26, 13:59
+
Jean-Marc Spaggiari 2012-12-19, 22:59
Copy link to this message
-
Re: recommended nodes
Yeah,
I couldn't argue against LVMs when talking with the system admins.
In terms of speed its noise because the CPUs are pretty efficient and unless you have more than 1 drive per physical core, you will end up saturating your disk I/O.

In terms of MapR, you want the raw disk. (But we're talking Apache)
On Dec 19, 2012, at 4:59 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]> wrote:

> Finally, it took me a while to run those tests because it was way
> longer than expected, but here are the results:
>
> http://www.spaggiari.org/bonnie.html
>
> LVM is not really slower than JBOD and not really taking more CPU. So
> I will say, if you have to choose between the 2, take the one you
> prefer. Personally, I prefer LVM because it's easy to configure.
>
> The big winner here is RAID0. It's WAY faster than anything else. But
> it's using twice the space... Your choice.
>
> I did not get a chance to test with the Ubuntu tool because it's not
> working with LVM drives.
>
> JM
>
> 2012/11/28, Michael Segel <[EMAIL PROTECTED]>:
>> Ok, just a caveat.
>>
>> I am discussing MapR as part of a complete response. As Mohit posted MapR
>> takes the raw device for their MapR File System.
>> They do stripe on their own within what they call a volume.
>>
>> But going back to Apache...
>> You can stripe drives, however I wouldn't recommend it. I don't think the
>> performance gains would really matter.
>> You're going to end up getting blocked first by disk i/o, then your
>> controller card, then your network... assuming 10GBe.
>>
>> With only 2 disks on an 8 core system, you will hit disk i/o first and then
>> you'll watch your CPU Wait I/O climb.
>>
>> HTH
>>
>> -Mike
>>
>> On Nov 28, 2012, at 7:28 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Hi Mike,
>>>
>>> Why not using LVM with MapR? Since LVM is reading from 2 drives almost
>>> at the same time, it should be better than RAID0 or a single drive,
>>> no?
>>>
>>> 2012/11/28, Michael Segel <[EMAIL PROTECTED]>:
>>>> Just a couple of things.
>>>>
>>>> I'm neutral on the use of LVMs. Some would point out that there's some
>>>> overhead, but on the flip side, it can make managing the machines
>>>> easier.
>>>> If you're using MapR, you don't want to use LVMs but raw devices.
>>>>
>>>> In terms of GC, its going to depend on the heap size and not the total
>>>> memory. With respect to HBase. ... MSLABS is the way to go.
>>>>
>>>>
>>>> On Nov 28, 2012, at 12:05 PM, Jean-Marc Spaggiari
>>>> <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>> Hi Gregory,
>>>>>
>>>>> I founs this about LVM:
>>>>> -> http://blog.andrew.net.au/2006/08/09
>>>>> ->
>>>>> http://www.phoronix.com/scan.php?page=article&item=fedora_15_lvm&num=2
>>>>>
>>>>> Seems that performances are still correct with it. I will most
>>>>> probably give it a try and bench that too... I have one new hard drive
>>>>> which should arrived tomorrow. Perfect timing ;)
>>>>>
>>>>>
>>>>>
>>>>> JM
>>>>>
>>>>> 2012/11/28, Mohit Anchlia <[EMAIL PROTECTED]>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Nov 28, 2012, at 9:07 AM, Adrien Mogenet <[EMAIL PROTECTED]>
>>>>>> wrote:
>>>>>>
>>>>>>> Does HBase really benefit from 64 GB of RAM since allocating too
>>>>>>> large
>>>>>>> heap
>>>>>>> might increase GC time ?
>>>>>>>
>>>>>> Benefit you get is from OS cache
>>>>>>> Another question : why not RAID 0, in order to aggregate disk
>>>>>>> bandwidth
>>>>>>> ?
>>>>>>> (and thus keep 3x replication factor)
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 28, 2012 at 5:58 PM, Michael Segel
>>>>>>> <[EMAIL PROTECTED]>wrote:
>>>>>>>
>>>>>>>> Sorry,
>>>>>>>>
>>>>>>>> I need to clarify.
>>>>>>>>
>>>>>>>> 4GB per physical core is a good starting point.
>>>>>>>> So with 2 quad core chips, that is going to be 32GB.
>>>>>>>>
>>>>>>>> IMHO that's a minimum. If you go with HBase, you will want more.
>>>>>>>> (Actually
>>>>>>>> you will need more.) The next logical jump would be to 48 or 64GB.
>>
+
Varun Sharma 2012-12-20, 21:22
+
Jean-Marc Spaggiari 2012-12-20, 21:26
+
Varun Sharma 2012-12-20, 21:37
+
Jean-Marc Spaggiari 2012-12-20, 22:07
+
Adrien Mogenet 2012-12-20, 22:11