Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - HBase insert slow


+
徐歷盛 2013-02-08, 16:23
+
Mesika, Asaf 2013-02-08, 16:38
Copy link to this message
-
Re: HBase insert slow
Mohammad Tariq 2013-02-08, 16:41
Could you please show your hbase-site.xml? Also, try the basic tweaks like
usign putlist instead of put, turning wal off(if it is fine with you) etc.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Fri, Feb 8, 2013 at 10:08 PM, Mesika, Asaf <[EMAIL PROTECTED]> wrote:

> What's the rate you're getting (MB/sec or lines/sec)?
> What's your cluster setup?
>
> On Feb 8, 2013, at 6:23 PM, 徐歷盛 wrote:
>
> > Hi,
> >    I recently suffered HBase insert slow issue (read from mysql, process
> > and then insert into HBase). The partial DEBUG log as below. It looks
> that
> > most of the time were spent near "Cached location" and "Got ping response
> > for sessionid". Is there any way to improve the performance ? Thanks
> >    I tried HTable and HTablePool, both showed the similiar result.
> >
> > [13/02/08 08:59:07:106][][DEBUG]Call: next 2
> > [13/02/08 08:59:11:619][][DEBUG]Cached location for
> > TABLE2_IDX01,,1355799947553.1a08e9f6c702b56d3dc839b83ce86124. is
> > 10.39.128.126:60020
> > ...
> > ...
> > [13/02/08 08:59:16:180][][DEBUG]Call: next 2
> > [13/02/08 08:59:19:298][][DEBUG]Got ping response for sessionid:
> > 0x13c4449defa5228 after 0ms
> >
> >
> > Detail Log as below
> >
> ==============================================================================================================> >
> > [13/02/08 08:58:25:863][][INFO]Initiating client connection,
> > connectString=SERVER1:2181 sessionTimeout=180000 watcher=hconnection
> > [13/02/08 08:58:25:866][][DEBUG]zookeeper.disableAutoWatchReset is false
> > [13/02/08 08:58:25:907][][INFO]Opening socket connection to server
> SERVER1/
> > 10.39.128.115:2181
> > [13/02/08 08:58:25:910][][INFO]Socket connection established to SERVER1/
> > 10.39.128.115:2181, initiating session
> > [13/02/08 08:58:25:912][][DEBUG]Session establishment request sent on
> > SERVER1/10.39.128.115:2181
> > [13/02/08 08:58:25:917][][INFO]Session establishment complete on server
> > SERVER1/10.39.128.115:2181, sessionid = 0x13c4449defa5228, negotiated
> > timeout = 40000
> >
> > [13/02/08 08:58:30:581][][DEBUG]Lookedup root region location,
> >
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1df2964
> ;
> > hsa=10.39.128.152:60020
> > [13/02/08 08:58:30:664][][DEBUG]The ping interval is60000ms.
> > [13/02/08 08:58:30:729][][DEBUG]Connecting to
> > 10.39.128.152/10.39.128.152:60020
> > [13/02/08 08:58:30:757][][DEBUG]IPC Client (47) connection to
> > 10.39.128.152/10.39.128.152:60020 from an unknown user sending #0
> > [13/02/08 08:58:30:827][][DEBUG]IPC Client (47) connection to
> > 10.39.128.152/10.39.128.152:60020 from an unknown user: starting, having
> > connections 1
> > [13/02/08 08:58:30:836][][DEBUG]IPC Client (47) connection to
> > 10.39.128.152/10.39.128.152:60020 from an unknown user got value #0
> > [13/02/08 08:58:30:837][][DEBUG]Call: getProtocolVersion 117
> > [13/02/08 08:58:30:838][][DEBUG]IPC Client (47) connection to
> > 10.39.128.152/10.39.128.152:60020 from an unknown user sending #1
> > [13/02/08 08:58:30:846][][DEBUG]IPC Client (47) connection to
> > 10.39.128.152/10.39.128.152:60020 from an unknown user got value #1
> > [13/02/08 08:58:30:848][][DEBUG]Call: getClosestRowBefore 10
> > [13/02/08 08:58:35:396][][DEBUG]Cached location for .META.,,1.1028785192
> is
> > 10.39.128.158:60020
> > [13/02/08 08:58:35:398][][DEBUG]Connecting to
> > 10.39.128.158/10.39.128.158:60020
> > [13/02/08 08:58:35:399][][DEBUG]IPC Client (47) connection to
> > 10.39.128.158/10.39.128.158:60020 from an unknown user sending #2
> > [13/02/08 08:58:35:399][][DEBUG]IPC Client (47) connection to
> > 10.39.128.158/10.39.128.158:60020 from an unknown user: starting, having
> > connections 2
> > [13/02/08 08:58:35:401][][DEBUG]IPC Client (47) connection to
> > 10.39.128.158/10.39.128.158:60020 from an unknown user got value #2
> > [13/02/08 08:58:35:401][][DEBUG]Call: getProtocolVersion 3
> > [13/02/08 08:58:35:417][][DEBUG]IPC Client (47) connection to
+
徐歷盛 2013-02-09, 01:49