Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Not able to contact META (trunk version) - Master and RS on same node


Copy link to this message
-
Not able to contact META (trunk version) - Master and RS on same node
Hi All

During some random testing i faced this issue in Trunk version.

Master and RS are on the same node and i have only one node.

I created some tables, disabled and dropped it.

The META was intact.

Again when i tried to create the table, the Meta could not be contacted
though in the same machine.
On restarting the master things became fine again. I am not able to
reproduce  this issue.  But for sometime nothing actually worked because
the META RS became unreachable.

{logs}
2013-04-03 17:21:21,675 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Deleting regions
from META
2013-04-03 17:21:21,679 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted from META, regions: [{NAME =>
'TestTable,,1365024046851.e5e94a1c4adc038e6517e45bcbbab9a9.', STARTKEY =>
'', ENDKEY => '', ENCODED => e5e94a1c4adc038e6517e45bcbbab9a9,}]
2013-04-03 17:21:21,683 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Archiving region
TestTable,,1365024046851.e5e94a1c4adc038e6517e45bcbbab9a9. from FS
2013-04-03 17:21:21,683 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
ARCHIVING region
hdfs://localhost:9010/hbase/.tmp/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9
------------
(This is where the table got deleted)
2013-04-03 17:21:21,705 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Deleted all region files in:
hdfs://localhost:9010/hbase/.tmp/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9
2013-04-03 17:21:21,706 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Table
'TestTable' archived!
2013-04-03 17:21:21,706 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Removing
'TestTable' descriptor.
2013-04-03 17:21:21,708 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Marking
'TestTable' as deleted.
2013-04-03 17:21:21,709 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Attempt to release table
write lock on :TestTable
2013-04-03 17:21:21,712 DEBUG
org.apache.hadoop.hbase.zookeeper.lock.ZKInterProcessLockBase: Successfully
released /hbase/table-lock/TestTable/write-master:600000000000002
2013-04-03 17:21:21,712 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Released table lock on
:TestTable
------
(Again started creating the table)

2013-04-03 17:25:23,955 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Attempt to acquire table
write lock on :TestTable for:C_M_CREATE_TABLE
2013-04-03 17:25:23,961 DEBUG
org.apache.hadoop.hbase.zookeeper.lock.ZKInterProcessLockBase: Successfully
acquired a lock for /hbase/table-lock/TestTable/write-master:600000000000000
2013-04-03 17:25:23,961 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Acquired table write lock
on :TestTable for:C_M_CREATE_TABLE
2013-04-03 17:25:23,963 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'TestTable,,'
2013-04-03 17:25:23,963 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'TestTable,,'
2013-04-03 17:25:52,928 DEBUG org.apache.hadoop.hbase.util.FSUtils:
hdfs://localhost:9010/hbase/.archive/TestTable/14ffdb436fae5eec5fc0ac0a8ca89c57/info/.links-86f33ddcb5db4f7ba91f543303491908
doesn't exist
2013-04-03 17:25:52,937 DEBUG org.apache.hadoop.hbase.util.FSUtils:
hdfs://localhost:9010/hbase/.archive/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9/info/.links-be7b133a187c42b29cdd83ee555c8905
doesn't exist
2013-04-03 17:25:52,940 DEBUG org.apache.hadoop.hbase.util.FSUtils:
hdfs://localhost:9010/hbase/.archive/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9/info/.links-f30d16210ff84957a8a6800538d6884e
doesn't exist
2013-04-03 17:26:05,031 DEBUG
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping
load balance as cluster has only one node.
2013-04-03 17:26:05,032 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2013-04-03 17:26:05,032 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''

..................
2013-04-03 17:41:05,032 DEBUG
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping
load balance as cluster has only one node.
2013-04-03 17:41:24,456 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=7, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:41:24,456 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=6, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:42:24,473 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=8, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:43:24,533 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=8, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:43:24,534 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=7, numRetries=100