Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hbase connection closed when query multiple complicated hql with hive+hbase integration


Copy link to this message
-
Hbase connection closed when query multiple complicated hql with hive+hbase integration
Cheng Su 2012-11-07, 03:07
Hi, all.

I have a hive + hbase integration cluster. I met a performance issue.
I executed some complicated hql, but only the first one is actually running.
The rests are showed "running" on the job track web ui, but task
attempt failed with exceptions below:

java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
closed
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
closed
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
... 9 more

So actually the rests are queuing.

I guess it might be because the count(?) of hbase handler is too
small,  which handle access request from hive.
So that when a big job occupied all handles the rests have to wait
until the handlers are released.

Is my assumption right? and what settings should I tuning?

Thanks.

--

Regards,
Cheng Su