Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Table deletion got stuck


Copy link to this message
-
Re: Table deletion got stuck
Are the times on the master and tablet server synched?  The load of n8<< on
the tablet server seems to occur after delete is waiting for it.

master.log : 27 11:48:04,332 [tableOps.CleanUp] DEBUG: Still waiting for
table to be deleted: n8 locationState: n8<<@(null,10.0.0.10:41000
[43b1b039a081368],null)
tserver.log : 27 11:52:25,220 [tabletserver.TabletServer] INFO : Loading
tablet n8<<
On Wed, Nov 28, 2012 at 10:44 AM, Lin XIAO <[EMAIL PROTECTED]> wrote:

> n8 was an empty table created through the shell.  Here are the logs on
> machine 10.0.0.10
>
> 27 11:52:25,220 [tabletserver.TabletServer] INFO : Loading tablet n8<<
> 27 11:52:25,221 [tabletserver.TabletServer] INFO :
> cloud9/10.0.0.10:41000: got assignment from master: n8<<
> 27 11:52:25,221 [tabletserver.TabletServer] DEBUG: Loading extent: n8<<
> 27 11:52:25,221 [tabletserver.TabletServer] DEBUG: verifying extent n8<<
> 27 11:52:25,223 [tabletserver.Tablet] DEBUG: Looking at metadata {n8<
> future:43b1b039a081368 [] 423355 false=10.0.0.10:41000, n8< srv:dir []
> 423354 false=/default_tablet, n8< srv:lock [] 423354
> false=masters/lock/zlock-0000000184$43b1b039a08ad85, n8< srv:time []
> 423354 false=M0, n8< ~tab:~pr [] 423354 false=}
> 27 11:52:25,223 [tabletserver.Tablet] DEBUG: got [] for logs for n8<<
> 27 11:52:25,230 [tabletserver.Tablet] TABLET_HIST: n8<< opened
>
> Thanks,
> Lin
>
> On Wed, Nov 28, 2012 at 8:55 AM, Keith Turner <[EMAIL PROTECTED]> wrote:
> > Can you look at the logs for tablet server 10.0.0.10 and see what was
> going
> > on with tablet n8<<?
> >
> > Keith
> >
> >
> > On Tue, Nov 27, 2012 at 6:20 PM, Lin XIAO <[EMAIL PROTECTED]> wrote:
> >>
> >> I've only went through the master log generated today for FAILED
> >> transactions.
> >> CreateTable operations failed because the table already exist while
> >> the DeleteTable failed because the table doesn't exist. I think the
> >> user run his hadoop jobs several times with same table names. If the
> >> table cannot be deleted, the following create operations will fail.
> >> I'm not sure why he tried to delete an non-existed table though.
> >>
> >> 27 04:52:16,547 [fate.Fate] WARN : Failed to execute Repo,
> >> tid=1f4c647a48c383a6
> >> ThriftTableOperationException(tableId:gf, tableName:, op:DELETE,
> >> type:NOTFOUND, description:Table does not exists)
> >> at
> >>
> org.apache.accumulo.server.master.tableOps.Utils.reserveTable(Utils.java:82)
> >> at
> >>
> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:224)
> >> at
> >>
> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:212)
> >> at
> >>
> org.apache.accumulo.server.master.tableOps.TraceRepo.isReady(TraceRepo.java:50)
> >> at
> >> org.apache.accumulo.server.fate.Fate$TransactionRunner.run(Fate.java:62)
> >> at
> >>
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> >> at java.lang.Thread.run(Thread.java:662)
> >> 27 04:52:16,564 [zookeeper.DistributedReadWriteLock] DEBUG: Removing
> >> lock entry 1 userData 1f4c647a48c383a6 lockType WRITE
> >> 27 04:52:16,569 [tableOps.Utils] INFO : table gf (1f4c647a48c383a6)
> >> unlocked for write
> >>
> >> A deleteTable operation succeeded after I killed the tablet server.
> >> Here are some related logs:
> >>
> >> 27 11:46:32,656 [zookeeper.DistributedReadWriteLock] INFO : Added lock
> >> entry 0 userData 12f1518e50918eaa lockType WRITE
> >> 27 11:46:32,658 [tableOps.Utils] INFO : table n8 (12f1518e50918eaa)
> >> locked for write operation: DELETE
> >> 27 11:46:32,660 [tables.TableManager] DEBUG: Transitioning state for
> >> table n8 from ONLINE to DELETING
> >> 27 11:46:32,662 [master.EventCoordinator] INFO : deleting table n8
> >> 27 11:46:32,663 [state.ZooTabletStateStore] DEBUG: Returning root
> >> tablet state:
> >> !0;!0<<@(null,10.0.0.36:41000[33b1b27a4ad120a],10.0.0.36:41000
> [33b1b27a4ad120a])
> >> 27 11:46:32,663 [tables.TableManager] DEBUG: State transition to
> >> DELETING @ WatchedEvent state:SyncConnected type:NodeDataChanged