Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> AssignmentManager looping?


Copy link to this message
-
AssignmentManager looping?
My master keep logging that:

2013-07-31 21:52:59,201 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,201 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,339 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,339 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,461 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,461 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,636 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,636 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,074 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,074 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,261 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,261 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,417 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,417 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split

hbase@node3:~/hbase-0.94.3$ cat logs/hbase-hbase-master-node3.log* | grep
"Region 270a9c371fcbe9cd9a04986e0b77d16b not found " | wc
   5042   65546  927728
Then crashed.
2013-07-31 22:22:46,072 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-07-31 22:22:46,073 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state : work_proposed,\x02\xE8\x92'\x00\x00\x00\x00
http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 .. Cannot
transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
work_proposed,\x02\xE8\x92'\x00\x00\x00\x00
http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 .. Cannot
transit it to OFFLINE.
    at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
    at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-07-31 22:22:46,075 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.HMaster$2:
node3,60000,1375322220614-BalancerChore exiting
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
node3,60000,1375322220614-CatalogJanitor exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apa
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB