Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Table deletion got stuck


Copy link to this message
-
Re: Table deletion got stuck
Can you look at the logs for tablet server 10.0.0.10 and see what was going
on with tablet n8<<?

Keith

On Tue, Nov 27, 2012 at 6:20 PM, Lin XIAO <[EMAIL PROTECTED]> wrote:

> I've only went through the master log generated today for FAILED
> transactions.
> CreateTable operations failed because the table already exist while
> the DeleteTable failed because the table doesn't exist. I think the
> user run his hadoop jobs several times with same table names. If the
> table cannot be deleted, the following create operations will fail.
> I'm not sure why he tried to delete an non-existed table though.
>
> 27 04:52:16,547 [fate.Fate] WARN : Failed to execute Repo,
> tid=1f4c647a48c383a6
> ThriftTableOperationException(tableId:gf, tableName:, op:DELETE,
> type:NOTFOUND, description:Table does not exists)
> at
> org.apache.accumulo.server.master.tableOps.Utils.reserveTable(Utils.java:82)
> at
> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:224)
> at
> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:212)
> at
> org.apache.accumulo.server.master.tableOps.TraceRepo.isReady(TraceRepo.java:50)
> at org.apache.accumulo.server.fate.Fate$TransactionRunner.run(Fate.java:62)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:662)
> 27 04:52:16,564 [zookeeper.DistributedReadWriteLock] DEBUG: Removing
> lock entry 1 userData 1f4c647a48c383a6 lockType WRITE
> 27 04:52:16,569 [tableOps.Utils] INFO : table gf (1f4c647a48c383a6)
> unlocked for write
>
> A deleteTable operation succeeded after I killed the tablet server.
> Here are some related logs:
>
> 27 11:46:32,656 [zookeeper.DistributedReadWriteLock] INFO : Added lock
> entry 0 userData 12f1518e50918eaa lockType WRITE
> 27 11:46:32,658 [tableOps.Utils] INFO : table n8 (12f1518e50918eaa)
> locked for write operation: DELETE
> 27 11:46:32,660 [tables.TableManager] DEBUG: Transitioning state for
> table n8 from ONLINE to DELETING
> 27 11:46:32,662 [master.EventCoordinator] INFO : deleting table n8
> 27 11:46:32,663 [state.ZooTabletStateStore] DEBUG: Returning root
> tablet state: !0;!0<<@(null,10.0.0.36:41000
> [33b1b27a4ad120a],10.0.0.36:41000[33b1b27a4ad120a])
> 27 11:46:32,663 [tables.TableManager] DEBUG: State transition to
> DELETING @ WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/accumulo/c8e02396-a69f-48be-aec2-045bbc55fa0c/tables/n8/state
> 27 11:46:32,663 [master.EventCoordinator] INFO : Table state in
> zookeeper changed for n8 to DELETING
> 27 11:46:32,685 [master.Master] DEBUG: Finished gathering information
> from 30 servers in 0.02 seconds
> 27 11:46:32,686 [master.Master] DEBUG: Telling
> 10.0.0.38:41000[23b1b03315212c6] to use loggers [10.0.0.64:11224,
> 10.0.0.15:11224]
> 27 11:46:32,686 [master.Master] DEBUG: Telling
> 10.0.0.36:41000[33b1b27a4ad120a] to use loggers [10.0.0.47:11224,
> 10.0.0.51:11224]
> 27 11:48:04,332 [tableOps.CleanUp] DEBUG: Still waiting for table to
> be deleted: n8 locationState:
> n8<<@(null,10.0.0.10:41000[43b1b039a081368],null)
> 27 14:26:09,101 [zookeeper.ZooReaderWriter] WARN : Error connecting to
> zookeeper, will retry in 250
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /accumulo/c8e02396-a69f-48be-aec2-045bbc55fa0c/fate/tx_12f1518e50918eaa
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:950)
>         at
> org.apache.accumulo.core.zookeeper.ZooReader.getData(ZooReader.java:42)
>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)