Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Table deletion got stuck


Copy link to this message
-
Re: Table deletion got stuck
Can you look at the logs for tablet server 10.0.0.10 and see what was going
on with tablet n8<<?

Keith

On Tue, Nov 27, 2012 at 6:20 PM, Lin XIAO <[EMAIL PROTECTED]> wrote:

> I've only went through the master log generated today for FAILED
> transactions.
> CreateTable operations failed because the table already exist while
> the DeleteTable failed because the table doesn't exist. I think the
> user run his hadoop jobs several times with same table names. If the
> table cannot be deleted, the following create operations will fail.
> I'm not sure why he tried to delete an non-existed table though.
>
> 27 04:52:16,547 [fate.Fate] WARN : Failed to execute Repo,
> tid=1f4c647a48c383a6
> ThriftTableOperationException(tableId:gf, tableName:, op:DELETE,
> type:NOTFOUND, description:Table does not exists)
> at
> org.apache.accumulo.server.master.tableOps.Utils.reserveTable(Utils.java:82)
> at
> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:224)
> at
> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:212)
> at
> org.apache.accumulo.server.master.tableOps.TraceRepo.isReady(TraceRepo.java:50)
> at org.apache.accumulo.server.fate.Fate$TransactionRunner.run(Fate.java:62)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:662)
> 27 04:52:16,564 [zookeeper.DistributedReadWriteLock] DEBUG: Removing
> lock entry 1 userData 1f4c647a48c383a6 lockType WRITE
> 27 04:52:16,569 [tableOps.Utils] INFO : table gf (1f4c647a48c383a6)
> unlocked for write
>
> A deleteTable operation succeeded after I killed the tablet server.
> Here are some related logs:
>
> 27 11:46:32,656 [zookeeper.DistributedReadWriteLock] INFO : Added lock
> entry 0 userData 12f1518e50918eaa lockType WRITE
> 27 11:46:32,658 [tableOps.Utils] INFO : table n8 (12f1518e50918eaa)
> locked for write operation: DELETE
> 27 11:46:32,660 [tables.TableManager] DEBUG: Transitioning state for
> table n8 from ONLINE to DELETING
> 27 11:46:32,662 [master.EventCoordinator] INFO : deleting table n8
> 27 11:46:32,663 [state.ZooTabletStateStore] DEBUG: Returning root
> tablet state: !0;!0<<@(null,10.0.0.36:41000
> [33b1b27a4ad120a],10.0.0.36:41000[33b1b27a4ad120a])
> 27 11:46:32,663 [tables.TableManager] DEBUG: State transition to
> DELETING @ WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/accumulo/c8e02396-a69f-48be-aec2-045bbc55fa0c/tables/n8/state
> 27 11:46:32,663 [master.EventCoordinator] INFO : Table state in
> zookeeper changed for n8 to DELETING
> 27 11:46:32,685 [master.Master] DEBUG: Finished gathering information
> from 30 servers in 0.02 seconds
> 27 11:46:32,686 [master.Master] DEBUG: Telling
> 10.0.0.38:41000[23b1b03315212c6] to use loggers [10.0.0.64:11224,
> 10.0.0.15:11224]
> 27 11:46:32,686 [master.Master] DEBUG: Telling
> 10.0.0.36:41000[33b1b27a4ad120a] to use loggers [10.0.0.47:11224,
> 10.0.0.51:11224]
> 27 11:48:04,332 [tableOps.CleanUp] DEBUG: Still waiting for table to
> be deleted: n8 locationState:
> n8<<@(null,10.0.0.10:41000[43b1b039a081368],null)
> 27 14:26:09,101 [zookeeper.ZooReaderWriter] WARN : Error connecting to
> zookeeper, will retry in 250
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /accumulo/c8e02396-a69f-48be-aec2-045bbc55fa0c/fate/tx_12f1518e50918eaa
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:950)
>         at
> org.apache.accumulo.core.zookeeper.ZooReader.getData(ZooReader.java:42)
>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB