Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo, mail # user - Table deletion got stuck


+
Lin XIAO 2012-11-27, 21:28
+
Keith Turner 2012-11-27, 21:38
+
Lin XIAO 2012-11-27, 21:42
+
Keith Turner 2012-11-27, 22:22
+
John Vines 2012-11-27, 22:24
+
Lin XIAO 2012-11-27, 23:20
+
Keith Turner 2012-11-28, 13:55
+
Lin XIAO 2012-11-28, 15:44
+
Keith Turner 2012-11-28, 15:55
Copy link to this message
-
Re: Table deletion got stuck
Lin XIAO 2012-11-28, 16:07
No. I think there were about 5 minutes delayed on the server. I didn't
realize that ntp wasn't running on the server until seeing the
problems.

On Wed, Nov 28, 2012 at 10:55 AM, Keith Turner <[EMAIL PROTECTED]> wrote:
> Are the times on the master and tablet server synched?  The load of n8<< on
> the tablet server seems to occur after delete is waiting for it.
>
> master.log : 27 11:48:04,332 [tableOps.CleanUp] DEBUG: Still waiting for
> table to be deleted: n8 locationState:
> n8<<@(null,10.0.0.10:41000[43b1b039a081368],null)
> tserver.log : 27 11:52:25,220 [tabletserver.TabletServer] INFO : Loading
> tablet n8<<
>
>
> On Wed, Nov 28, 2012 at 10:44 AM, Lin XIAO <[EMAIL PROTECTED]> wrote:
>>
>> n8 was an empty table created through the shell.  Here are the logs on
>> machine 10.0.0.10
>>
>> 27 11:52:25,220 [tabletserver.TabletServer] INFO : Loading tablet n8<<
>> 27 11:52:25,221 [tabletserver.TabletServer] INFO :
>> cloud9/10.0.0.10:41000: got assignment from master: n8<<
>> 27 11:52:25,221 [tabletserver.TabletServer] DEBUG: Loading extent: n8<<
>> 27 11:52:25,221 [tabletserver.TabletServer] DEBUG: verifying extent n8<<
>> 27 11:52:25,223 [tabletserver.Tablet] DEBUG: Looking at metadata {n8<
>> future:43b1b039a081368 [] 423355 false=10.0.0.10:41000, n8< srv:dir []
>> 423354 false=/default_tablet, n8< srv:lock [] 423354
>> false=masters/lock/zlock-0000000184$43b1b039a08ad85, n8< srv:time []
>> 423354 false=M0, n8< ~tab:~pr [] 423354 false=}
>> 27 11:52:25,223 [tabletserver.Tablet] DEBUG: got [] for logs for n8<<
>> 27 11:52:25,230 [tabletserver.Tablet] TABLET_HIST: n8<< opened
>>
>> Thanks,
>> Lin
>>
>> On Wed, Nov 28, 2012 at 8:55 AM, Keith Turner <[EMAIL PROTECTED]> wrote:
>> > Can you look at the logs for tablet server 10.0.0.10 and see what was
>> > going
>> > on with tablet n8<<?
>> >
>> > Keith
>> >
>> >
>> > On Tue, Nov 27, 2012 at 6:20 PM, Lin XIAO <[EMAIL PROTECTED]> wrote:
>> >>
>> >> I've only went through the master log generated today for FAILED
>> >> transactions.
>> >> CreateTable operations failed because the table already exist while
>> >> the DeleteTable failed because the table doesn't exist. I think the
>> >> user run his hadoop jobs several times with same table names. If the
>> >> table cannot be deleted, the following create operations will fail.
>> >> I'm not sure why he tried to delete an non-existed table though.
>> >>
>> >> 27 04:52:16,547 [fate.Fate] WARN : Failed to execute Repo,
>> >> tid=1f4c647a48c383a6
>> >> ThriftTableOperationException(tableId:gf, tableName:, op:DELETE,
>> >> type:NOTFOUND, description:Table does not exists)
>> >> at
>> >>
>> >> org.apache.accumulo.server.master.tableOps.Utils.reserveTable(Utils.java:82)
>> >> at
>> >>
>> >> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:224)
>> >> at
>> >>
>> >> org.apache.accumulo.server.master.tableOps.DeleteTable.isReady(DeleteTable.java:212)
>> >> at
>> >>
>> >> org.apache.accumulo.server.master.tableOps.TraceRepo.isReady(TraceRepo.java:50)
>> >> at
>> >>
>> >> org.apache.accumulo.server.fate.Fate$TransactionRunner.run(Fate.java:62)
>> >> at
>> >>
>> >> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>> >> at java.lang.Thread.run(Thread.java:662)
>> >> 27 04:52:16,564 [zookeeper.DistributedReadWriteLock] DEBUG: Removing
>> >> lock entry 1 userData 1f4c647a48c383a6 lockType WRITE
>> >> 27 04:52:16,569 [tableOps.Utils] INFO : table gf (1f4c647a48c383a6)
>> >> unlocked for write
>> >>
>> >> A deleteTable operation succeeded after I killed the tablet server.
>> >> Here are some related logs:
>> >>
>> >> 27 11:46:32,656 [zookeeper.DistributedReadWriteLock] INFO : Added lock
>> >> entry 0 userData 12f1518e50918eaa lockType WRITE
>> >> 27 11:46:32,658 [tableOps.Utils] INFO : table n8 (12f1518e50918eaa)
>> >> locked for write operation: DELETE
>> >> 27 11:46:32,660 [tables.TableManager] DEBUG: Transitioning state for
>> >> table n8 from ONLINE to DELETING
>> >> 27 11:46:32,662 [master.EventCoordinator] INFO : deleting table n8
+
Keith Turner 2012-11-28, 16:20
+
Lin XIAO 2012-11-28, 19:22
+
Keith Turner 2012-11-29, 18:16
+
Keith Turner 2012-11-28, 16:08
+
Lin XIAO 2012-11-28, 19:05
+
Keith Turner 2012-11-28, 16:56