Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Pipes Property Not Passed In


Copy link to this message
-
Pipes Property Not Passed In
I implemented my own InputFormat/RecordReader, and I try to run it with
Hadoop Pipes. I understand I could pass in properties to Pipes program by
either:

<property>
        <name>hadoop.pipes.java.recordreader</name>
        <value>false</value>
</property>

or alterntively "-D hadoop.pipes.java.recordreader=false".

However, when I ran with above configuration and with my record reader, I
got error
Hadoop Pipes Exception: RecordReader defined when not needed. at
impl/HadoopPipes.cc:798 in virtual void
HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)

It pipes did not seem to pick up my setting of
hadoop.pipes.java.recordreader as false.

I've tried using conf.xml or putting -D or the combine of both. None
worked. I've googled the whole day but could not find the reason. Did I
miss something here?

I am using hadoop-1.0.4.

Here is my conf.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>hadoop.pipes.executable</name>
    <value>bin/cpc</value>
  </property>
  <property>
    <name>hadoop.pipes.java.recordreader</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.pipes.java.recordwriter</name>
    <value>true</value>
  </property>
</configuration>

Here is the command

$HADOOP pipes \
-conf $CONF \
-files 0 \
-libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
-input $IN \
-output $OUT \
-program bin/$NAME \
-reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
-inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat

where $CONF is full path to conf.xml

I could provide more info if that hellps to determine the reason.