Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Re: HConnectionImplementation.listTables and list table exception


Copy link to this message
-
Re: HConnectionImplementation.listTables and list table exception
Hello Michelle,
   How many regions totally are there in your 600 nodes cluster? Looks like many of them are pending for open and being assigned to region servers.
Can you see many items under zookeeper dir /hbase/unassigned?

You would like to refer http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html?

Best Regards, Julian
On Oct 23, 2013, at 01:46 PM, 张莉苹 <[EMAIL PROTECTED]> wrote:

> Dear HBase dev and users,
>
> Did you meet this
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> issue?
>
> We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> but it seemed hbase master was busy handling transition with zookeeper, and
> hbase “list” could not get response. The hbase table was created but it
> didn't do any insert.
>
> Do you have any idea of the root cause and how to fix it? :)Highly
> appreciate for your answers!
>
>
>
> Here is the exception stack:
> ---------------------------------------------------
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy7.getHTableDescriptors(Unknown Source)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>
>
>
>
> hbase master log:
>
> -----------------------------
>
> 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,279 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> ic.analyticsworkbench.com,60020,1381883086785,
> region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,280 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0898*.ic.analyticsworkbench.com,60020,1381883200494,
> region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,289 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0693.ic.analyticsworkbench.com,60020,1381881773670,
> region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late
>
>
>
> We also try to "list" in hbase shell,it also failed:
>
> The hbase “list” got error as:
>
> ------------------------------------------
>
>
>
> hbase(main):001:0> list
>
> TABLE
>
>
>
>
> ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote> node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)