Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hbase connection closed when query multiple complicated hql with hive+hbase integration

Copy link to this message
Re: Hbase connection closed when query multiple complicated hql with hive+hbase integration
The exceptions seem to be another problem.
They all happened on one node.
And after the task attempts failed at that node,
retried on other nodes and no exceptions.
So that, the exception maybe have nothing to do with the performance issue.

On Wed, Nov 7, 2012 at 11:07 AM, Cheng Su <[EMAIL PROTECTED]> wrote:
> Hi, all.
> I have a hive + hbase integration cluster. I met a performance issue.
> I executed some complicated hql, but only the first one is actually running.
> The rests are showed "running" on the job track web ui, but task
> attempt failed with exceptions below:
> java.io.IOException: java.io.IOException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
> closed
>         at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>         at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>         at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
>         at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
>         at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Unknown Source)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.io.IOException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
> closed
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
>         at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
>         at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
>         ... 9 more
> So actually the rests are queuing.
> I guess it might be because the count(?) of hbase handler is too
> small,  which handle access request from hive.
> So that when a big job occupied all handles the rests have to wait
> until the handlers are released.
> Is my assumption right? and what settings should I tuning?
> Thanks.
> --
> Regards,
> Cheng Su


Cheng Su