Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> AssignmentManager looping?


Copy link to this message
-
AssignmentManager looping?
My master keep logging that:

2013-07-31 21:52:59,201 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,201 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,339 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,339 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,461 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,461 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,636 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,636 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,074 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,074 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,261 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,261 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,417 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,417 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split

hbase@node3:~/hbase-0.94.3$ cat logs/hbase-hbase-master-node3.log* | grep
"Region 270a9c371fcbe9cd9a04986e0b77d16b not found " | wc
   5042   65546  927728
Then crashed.
2013-07-31 22:22:46,072 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-07-31 22:22:46,073 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state : work_proposed,\x02\xE8\x92'\x00\x00\x00\x00
http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 .. Cannot
transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
work_proposed,\x02\xE8\x92'\x00\x00\x00\x00
http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 .. Cannot
transit it to OFFLINE.
    at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
    at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-07-31 22:22:46,075 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.HMaster$2:
node3,60000,1375322220614-BalancerChore exiting
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
node3,60000,1375322220614-CatalogJanitor exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apa