-Pipes Property Not Passed In
Xun TANG 2013-04-24, 05:14
I implemented my own InputFormat/RecordReader, and I try to run it with
Hadoop Pipes. I understand I could pass in properties to Pipes program by
or alterntively "-D hadoop.pipes.java.recordreader=false".
However, when I ran with above configuration and with my record reader, I
Hadoop Pipes Exception: RecordReader defined when not needed. at
impl/HadoopPipes.cc:798 in virtual void
HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)
It pipes did not seem to pick up my setting of
hadoop.pipes.java.recordreader as false.
I've tried using conf.xml or putting -D or the combine of both. None
worked. I've googled the whole day but could not find the reason. Did I
miss something here?
I am using hadoop-1.0.4.
Here is my conf.xml
Here is the command
$HADOOP pipes \
-conf $CONF \
-files 0 \
-libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
-input $IN \
-output $OUT \
-program bin/$NAME \
-reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
where $CONF is full path to conf.xml
I could provide more info if that hellps to determine the reason.