Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Not able to contact META (trunk version) - Master and RS on same node


Copy link to this message
-
Not able to contact META (trunk version) - Master and RS on same node
Hi All

During some random testing i faced this issue in Trunk version.

Master and RS are on the same node and i have only one node.

I created some tables, disabled and dropped it.

The META was intact.

Again when i tried to create the table, the Meta could not be contacted
though in the same machine.
On restarting the master things became fine again. I am not able to
reproduce  this issue.  But for sometime nothing actually worked because
the META RS became unreachable.

{logs}
2013-04-03 17:21:21,675 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Deleting regions
from META
2013-04-03 17:21:21,679 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted from META, regions: [{NAME =>
'TestTable,,1365024046851.e5e94a1c4adc038e6517e45bcbbab9a9.', STARTKEY =>
'', ENDKEY => '', ENCODED => e5e94a1c4adc038e6517e45bcbbab9a9,}]
2013-04-03 17:21:21,683 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Archiving region
TestTable,,1365024046851.e5e94a1c4adc038e6517e45bcbbab9a9. from FS
2013-04-03 17:21:21,683 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
ARCHIVING region
hdfs://localhost:9010/hbase/.tmp/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9
------------
(This is where the table got deleted)
2013-04-03 17:21:21,705 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Deleted all region files in:
hdfs://localhost:9010/hbase/.tmp/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9
2013-04-03 17:21:21,706 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Table
'TestTable' archived!
2013-04-03 17:21:21,706 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Removing
'TestTable' descriptor.
2013-04-03 17:21:21,708 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Marking
'TestTable' as deleted.
2013-04-03 17:21:21,709 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Attempt to release table
write lock on :TestTable
2013-04-03 17:21:21,712 DEBUG
org.apache.hadoop.hbase.zookeeper.lock.ZKInterProcessLockBase: Successfully
released /hbase/table-lock/TestTable/write-master:600000000000002
2013-04-03 17:21:21,712 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Released table lock on
:TestTable
------
(Again started creating the table)

2013-04-03 17:25:23,955 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Attempt to acquire table
write lock on :TestTable for:C_M_CREATE_TABLE
2013-04-03 17:25:23,961 DEBUG
org.apache.hadoop.hbase.zookeeper.lock.ZKInterProcessLockBase: Successfully
acquired a lock for /hbase/table-lock/TestTable/write-master:600000000000000
2013-04-03 17:25:23,961 DEBUG
org.apache.hadoop.hbase.master.TableLockManager: Acquired table write lock
on :TestTable for:C_M_CREATE_TABLE
2013-04-03 17:25:23,963 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'TestTable,,'
2013-04-03 17:25:23,963 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'TestTable,,'
2013-04-03 17:25:52,928 DEBUG org.apache.hadoop.hbase.util.FSUtils:
hdfs://localhost:9010/hbase/.archive/TestTable/14ffdb436fae5eec5fc0ac0a8ca89c57/info/.links-86f33ddcb5db4f7ba91f543303491908
doesn't exist
2013-04-03 17:25:52,937 DEBUG org.apache.hadoop.hbase.util.FSUtils:
hdfs://localhost:9010/hbase/.archive/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9/info/.links-be7b133a187c42b29cdd83ee555c8905
doesn't exist
2013-04-03 17:25:52,940 DEBUG org.apache.hadoop.hbase.util.FSUtils:
hdfs://localhost:9010/hbase/.archive/TestTable/e5e94a1c4adc038e6517e45bcbbab9a9/info/.links-f30d16210ff84957a8a6800538d6884e
doesn't exist
2013-04-03 17:26:05,031 DEBUG
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping
load balance as cluster has only one node.
2013-04-03 17:26:05,032 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2013-04-03 17:26:05,032 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''

..................
2013-04-03 17:41:05,032 DEBUG
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping
load balance as cluster has only one node.
2013-04-03 17:41:24,456 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=7, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:41:24,456 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=6, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:42:24,473 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=8, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:43:24,533 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=8, numRetries=100 message=Call to
ram.sh.intel.com/10.239.47.144:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.239.47.144:59884 remoteram.sh.intel.com/10.239.47.144:60020]
2013-04-03 17:43:24,534 WARN org.apache.hadoop.hbase.client.ServerCallable:
Received exception, tries=7, numRetries=100
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB