Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Table deletion got stuck


Copy link to this message
-
Re: Table deletion got stuck
I've only went through the master log generated today for FAILED transactions.
CreateTable operations failed because the table already exist while
the DeleteTable failed because the table doesn't exist. I think the
user run his hadoop jobs several times with same table names. If the
table cannot be deleted, the following create operations will fail.
I'm not sure why he tried to delete an non-existed table though.

27 04:52:16,547 [fate.Fate] WARN : Failed to execute Repo, tid=1f4c647a48c383a6
ThriftTableOperationException(tableId:gf, tableName:, op:DELETE,
type:NOTFOUND, description:Table does not exists)
at org.apache.accumulo.server.master.tableOps.Utils.reserveTable(Utils.java:82)
at org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:224)
at org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:212)
at org.apache.accumulo.server.master.tableOps.TraceRepo.isReady(TraceRepo.java:50)
at org.apache.accumulo.server.fate.Fate$TransactionRunner.run(Fate.java:62)
at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:662)
27 04:52:16,564 [zookeeper.DistributedReadWriteLock] DEBUG: Removing
lock entry 1 userData 1f4c647a48c383a6 lockType WRITE
27 04:52:16,569 [tableOps.Utils] INFO : table gf (1f4c647a48c383a6)
unlocked for write

A deleteTable operation succeeded after I killed the tablet server.
Here are some related logs:

27 11:46:32,656 [zookeeper.DistributedReadWriteLock] INFO : Added lock
entry 0 userData 12f1518e50918eaa lockType WRITE
27 11:46:32,658 [tableOps.Utils] INFO : table n8 (12f1518e50918eaa)
locked for write operation: DELETE
27 11:46:32,660 [tables.TableManager] DEBUG: Transitioning state for
table n8 from ONLINE to DELETING
27 11:46:32,662 [master.EventCoordinator] INFO : deleting table n8
27 11:46:32,663 [state.ZooTabletStateStore] DEBUG: Returning root
tablet state: !0;!0<<@(null,10.0.0.36:41000[33b1b27a4ad120a],10.0.0.36:41000[33b1b27a4ad120a])
27 11:46:32,663 [tables.TableManager] DEBUG: State transition to
DELETING @ WatchedEvent state:SyncConnected type:NodeDataChanged
path:/accumulo/c8e02396-a69f-48be-aec2-045bbc55fa0c/tables/n8/state
27 11:46:32,663 [master.EventCoordinator] INFO : Table state in
zookeeper changed for n8 to DELETING
27 11:46:32,685 [master.Master] DEBUG: Finished gathering information
from 30 servers in 0.02 seconds
27 11:46:32,686 [master.Master] DEBUG: Telling
10.0.0.38:41000[23b1b03315212c6] to use loggers [10.0.0.64:11224,
10.0.0.15:11224]
27 11:46:32,686 [master.Master] DEBUG: Telling
10.0.0.36:41000[33b1b27a4ad120a] to use loggers [10.0.0.47:11224,
10.0.0.51:11224]
27 11:48:04,332 [tableOps.CleanUp] DEBUG: Still waiting for table to
be deleted: n8 locationState:
n8<<@(null,10.0.0.10:41000[43b1b039a081368],null)
27 14:26:09,101 [zookeeper.ZooReaderWriter] WARN : Error connecting to
zookeeper, will retry in 250
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/accumulo/c8e02396-a69f-48be-aec2-045bbc55fa0c/fate/tx_12f1518e50918eaa
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:950)
at org.apache.accumulo.core.zookeeper.ZooReader.getData(ZooReader.java:42)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:169)
at $Proxy3.getData(Unknown Source)
at org.apache.accumulo.server.fate.ZooStore._getStatus(ZooStore.java:305)
at org.apache.accumulo.server.fate.ZooStore.waitForStatusChange(ZooStore.java:327)
at org.apache.accumulo.server.fate.Fate.waitForCompletion(Fate.java:186)
at org.apache.accumulo.server.master.Master$MasterClientServiceHandler.waitForTableOperation(Master.java:1041)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
at $Proxy4.waitForTableOperation(Unknown Source)
at org.apache.accumulo.core.master.thrift.MasterClientService$Processor$waitForTableOperation.process(MasterClientService.java:1988)
at org.apache.accumulo.core.master.thrift.MasterClientService$Processor.process(MasterClientService.java:1456)
at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:662)
27 16:08:04,530 [master.CoordinateRecoveryTask] WARN : Recovery of
10.0.0.10:11224:d5706280-f28b-4da0-9a64-8262b946686a failed
27 16:08:04,532 [master.CoordinateRecoveryTask] DEBUG: Starting log
recovery: 10.0.0.14:11224/9887ab7c-6352-43f9-b6bd-abb901aa40b3
27 16:08:04,533 [master.CoordinateRecoveryTask] DEBUG: Starting to
copy 9887ab7c-6352-43f9-b6bd-abb901aa40b3 from 10.0.0.14:11224
27 16:08:04,673 [security.Auditor] AUDIT: Using credentials !SYSTEM:
deleted table n8
27 16:08:04,674 [zookeeper.DistributedReadWriteLock] DEBUG: Removing
lock entry 0 userData 12f1518e50918eaa lockType WRITE
27 16:08:04,678 [tableOps.Utils] INFO : table n8 (12f1518e50918eaa)
unlocked for write
27 16:08:04,678 [tabl
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB