Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # user >> _SUCCESS files appearing in demuxOutput


+
Corbin Hoenes 2011-02-24, 18:12
+
Eric Yang 2011-02-24, 18:55
Copy link to this message
-
Re: _SUCCESS files appearing in demuxOutput
This filename is coming from here: http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/constant-values.html
In general for hadoop you may want to avoid looking at any "_*" file since those are Hadoop related files like (_temporary, _log,…)

/Jerome.
From: Eric Yang <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Thu, 24 Feb 2011 10:55:57 -0800
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: _SUCCESS files appearing in demuxOutput

Hi Corbin,

I have not seen this.  What is the version of hadoop that you are using, are you using 0.21?  It looks like the _SUCCESS file is spill out after demux mapreduce job.  There are two possibilities leading to the creation of this file.  Demux is modified and it is doing something that is unexpected, or the mapreduce framework 0.21 put that file there.
If you are using 0.21, I would recommend to avoid it.

A more stable version of Hadoop is 0.20.100 branch, and you can download it from:

http://people.apache.org/~eyang/

Regards,
Eric

On 2/24/11 10:12 AM, "Corbin Hoenes" <[EMAIL PROTECTED]> wrote:

Anyone seen this?

/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS

I clean them out and I keep getting the same file showing up and chukwa doesn't know how to handle it:

postProcess.log:
2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster (_SUCCESS)
2011-02-21 06:51:55,027 INFO main MoveToRepository - processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS)
2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in processDemuxOutput:
java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS is not a directory!
    at org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.processClutserDirectory(MoveToRepository.java:54)
    at org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(MoveToRepository.java:250)
    at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movetoMainRepository(PostProcessorManager.java:201)
    at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start(PostProcessorManager.java:146)
    at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(PostProcessorManager.java:80)
+
Corbin Hoenes 2011-02-25, 00:24
+
Ariel Rabkin 2011-02-25, 00:27
+
Eric Yang 2011-02-25, 00:32
+
Jerome Boulon 2011-02-25, 00:37
+
James Seigel 2011-02-25, 01:36