Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> HDFS file disappeared


Copy link to this message
-
Re: HDFS file disappeared
>Would be great if someone wrote some tools that, given a block ID, tracked
 >the life of the file that contained it (including renames of containing
 > dirs, etc). Shouldn't be too difficult.

There's a tool for this in MapRed's contrib section under
block_forensics.  It was released in 21, I believe.  It hasn't been kept
up to date though, so I'm not sure how functional it still is.
-Jakob

Todd Lipcon wrote:
> What's the last audit log entry prior to 2010-11-10 21:42:33,389?
>
> -Todd
>
> On Thu, Nov 11, 2010 at 2:10 PM, David Rosenstrauch <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Saw a couple more references to that block before the "to delete
>      blk" messages:
>
>     2010-11-10 21:42:33,389 INFO org.apache.hadoop.hdfs.StateChange:
>     BLOCK* NameSystem.addToInvalidates: blk_-4237880568969698703 is
>     added to invalidSet of <our ip prefix>.169:50010
>     2010-11-10 21:42:33,389 INFO org.apache.hadoop.hdfs.StateChange:
>     BLOCK* NameSystem.addToInvalidates: blk_-4237880568969698703 is
>     added to invalidSet of <our ip prefix>.173:50010
>     2010-11-10 21:42:33,389 INFO org.apache.hadoop.hdfs.StateChange:
>     BLOCK* NameSystem.addToInvalidates: blk_-4237880568969698703 is
>     added to invalidSet of <our ip prefix>.176:50010
>
>     Again, I'm not sure why this is happening though.
>
>
>     BTW, I appreciate your comments below (about it getting moved out of
>     the temp directory and then getting removed in another pass).  But I
>     grepped the logs as you suggested, and I still don't see how it got
>     moved/deleted:
>
>     [root@hdmaster hadoop-0.20]# grep
>     _attempt_201010221550_0418_r_000001_0
>     hadoop-hadoop-namenode-hdmaster.log.2010-11-10
>     2010-11-10 21:42:28,442 INFO
>     org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
>     ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/<our ip
>     root>.176     cmd=create      src=<our root
>     dir>/2010.11.10-21.05.29/output/_temporary/_attempt_201010221550_0418_r_000001_0/shard2/IntentTrait.state
>      dst=null        perm=root:supergroup:rw-r--r--
>
>     2010-11-10 21:42:29,802 INFO org.apache.hadoop.hdfs.StateChange:
>     BLOCK* NameSystem.allocateBlock: <our root
>     dir>/2010.11.10-21.05.29/output/_temporary/_attempt_201010221550_0418_r_000001_0/shard2/IntentTrait.state.
>     blk_-4237880568969698703_13404582
>     2010-11-10 21:42:30,360 INFO org.apache.hadoop.hdfs.StateChange:
>     Removing lease on  file <our root
>     dir>/2010.11.10-21.05.29/output/_temporary/_attempt_201010221550_0418_r_000001_0/shard2/IntentTrait.state
>     from client DFSClient_attempt_201010221550_0418_r_000001_0
>     2010-11-10 21:42:30,360 INFO org.apache.hadoop.hdfs.StateChange:
>     DIR* NameSystem.completeFile: file <our root
>     dir>/2010.11.10-21.05.29/output/_temporary/_attempt_201010221550_0418_r_000001_0/shard2/IntentTrait.state
>     is closed by DFSClient_attempt_201010221550_0418_r_000001_0
>
>
>     I have to say, I'm really quite stumped.  Been looking at this all
>     afternoon, and I still have no idea how/why that file got purged.  :-(
>
>     Thanks,
>
>     DR
>
>
>     On 11/11/2010 02:13 PM, Todd Lipcon wrote:
>
>         Given that it's an MR output, my guess is it got moved out of
>         the temporary
>         directory when the job "Committed" and then was removed as
>         another pass. I'd
>         grep for the containing directory name in the audit logs to see
>         where it got
>         moved to and how it was eventually deleted.
>
>         Would be great if someone wrote some tools that, given a block
>         ID, tracked
>         the life of the file that contained it (including renames of
>         containing
>         dirs, etc). Shouldn't be too difficult.
>
>         -Todd
>
>         On Thu, Nov 11, 2010 at 9:38 AM, David
>         Rosenstrauch<[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>wrote:
>
>             Sorry, I stand corrected.  When I grep the name node logs
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB