Xun TANG 2013-04-24, 05:14
-Re: Pipes Property Not Passed In
Xun TANG 2013-04-24, 17:18
I think I pinned down the problem, but still can't find a solution.
It seems like 'hadoop.pipes.java.recordreader' is not the correct property
name to use?
Because I tried setting other properties (e.g. hadoop.pipes.executable) in
conf.xml and they were effectively passed in to pipes when the program
Where could I find the source code of pipes reading conf.xml file? Where
could I find a list of all properties (for pipes) and their default values?
Best I could find is this
but it does not list all properties.
On Tue, Apr 23, 2013 at 10:14 PM, Xun TANG <[EMAIL PROTECTED]> wrote:
> I implemented my own InputFormat/RecordReader, and I try to run it with
> Hadoop Pipes. I understand I could pass in properties to Pipes program by
> or alterntively "-D hadoop.pipes.java.recordreader=false".
> However, when I ran with above configuration and with my record reader, I
> got error
> Hadoop Pipes Exception: RecordReader defined when not needed. at
> impl/HadoopPipes.cc:798 in virtual void
> HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)
> It pipes did not seem to pick up my setting of
> hadoop.pipes.java.recordreader as false.
> I've tried using conf.xml or putting -D or the combine of both. None
> worked. I've googled the whole day but could not find the reason. Did I
> miss something here?
> I am using hadoop-1.0.4.
> Here is my conf.xml
> <?xml version="1.0"?>
> Here is the command
> $HADOOP pipes \
> -conf $CONF \
> -files 0 \
> -libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
> -input $IN \
> -output $OUT \
> -program bin/$NAME \
> -reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
> -inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat
> where $CONF is full path to conf.xml
> I could provide more info if that hellps to determine the reason.