Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - HBase read perfomnance and HBase client


Copy link to this message
-
Re: HBase read perfomnance and HBase client
Michael Segel 2013-08-01, 19:10
Ok... Bonded 1GbE is less than 2GbE, not sure of actual max throughput.

Are you hitting data in cache or are you fetching data from disk?
I mean can we rule out disk I/O because the data would most likely be in cache?

Are you monitoring your cluster w Ganglia? What do you see in terms of network traffic?
Are all of the nodes in the test cluster on the same switch? Including the client?
(Sorry, I'm currently looking at a network problem so now everything I see may be a networking problem. And a guy from Arista found me after our meetup last night so I am thinking about the impact on networking in the ecosystem. :-).  )
-Just some guy out in left field...

Sent from a remote device. Please excuse any typos...

Mike Segel

On Aug 1, 2013, at 1:11 PM, "Vladimir Rodionov" <[EMAIL PROTECTED]> wrote:

> 2x1Gb bonded, I think. This is our standard config.
>
>
> On Thu, Aug 1, 2013 at 10:27 AM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> Network? 1GbE or 10GbE?
>>
>> Sent from a remote device. Please excuse any typos...
>>
>> Mike Segel
>>
>> On Jul 31, 2013, at 9:27 PM, "Vladimir Rodionov" <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Some final numbers :
>>>
>>> Test config:
>>>
>>> HBase 0.94.6
>>> blockcache=true, block size = 64K, KV size = 62 bytes (raw).
>>>
>>> 5 Clients: 96GB, 16(32) CPUs (2.2Ghz), CentOS 5.7
>>> 1 RS Server: the same config.
>>>
>>> Local network with ping between hosts: 0.1 ms
>>>
>>>
>>> 1. HBase client hits the wall at ~ 50K per sec regardless of # of CPU,
>>> threads, IO pool size and other settings.
>>> 2. HBase server was able to sustain 170K per sec (with 64K block size).
>> All
>>> from block cache. KV size = 62 bytes (very small). This is for single Get
>>> op, 60 threads per client, 5 clients (on different hosts)
>>> 3. Multi - get hits the wall at the same 170K-200K per sec. Batch size
>>> tested: 30, 100. The same performance absolutely as with batch size = 1.
>>> Multi get has some internal issues on RegionServer side. May be excessive
>>> locking or some thing else.
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jul 30, 2013 at 2:01 PM, Vladimir Rodionov
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>> 1. SCR are enabled
>>>> 2. Single Configuration for all table did not work well, but I will try
>> it
>>>> again
>>>> 3. With Nagel I had 0.8ms avg, w/o - 0.4ms - I see the difference
>>>>
>>>>
>>>> On Tue, Jul 30, 2013 at 1:50 PM, lars hofhansl <[EMAIL PROTECTED]>
>> wrote:
>>>>
>>>>> With Nagle's you'd see something around 40ms. You are not saying 0.8ms
>>>>> RTT is bad, right? Are you seeing ~40ms latencies?
>>>>>
>>>>> This thread has gotten confusing.
>>>>>
>>>>> I would try these:
>>>>> * one Configuration for all tables. Or even use a single
>>>>> HConnection/Threadpool and use the HTable(byte[], HConnection,
>>>>> ExecutorService) constructor
>>>>> * disable Nagle's: set both ipc.server.tcpnodelay and
>>>>> hbase.ipc.client.tcpnodelay to true in hbase-site.xml (both client
>> *and*
>>>>> server)
>>>>> * increase hbase.client.ipc.pool.size in client's hbase-site.xml
>>>>> * enable short circuit reads (details depend on exact version of
>> Hadoop).
>>>>> Google will help :)
>>>>>
>>>>> -- Lars
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>> From: Vladimir Rodionov <[EMAIL PROTECTED]>
>>>>> To: [EMAIL PROTECTED]
>>>>> Cc:
>>>>> Sent: Tuesday, July 30, 2013 1:30 PM
>>>>> Subject: Re: HBase read perfomnance and HBase client
>>>>>
>>>>> This hbase.ipc.client.tcpnodelay (default - false) explains poor single
>>>>> thread performance and high latency ( 0.8ms in local network)?
>>>>>
>>>>>
>>>>> On Tue, Jul 30, 2013 at 1:22 PM, Vladimir Rodionov
>>>>> <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> One more observation: One Configuration instance per HTable gives 50%
>>>>>> boost as compared to single Configuration object for all HTable's -
>> from
>>>>>> 20K to 30K
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 30, 2013 at 1:17 PM, Vladimir Rodionov <
>>>>> [EMAIL PROTECTED]