Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Pipes Property Not Passed In


Copy link to this message
-
Pipes Property Not Passed In
I implemented my own InputFormat/RecordReader, and I try to run it with
Hadoop Pipes. I understand I could pass in properties to Pipes program by
either:

<property>
        <name>hadoop.pipes.java.recordreader</name>
        <value>false</value>
</property>

or alterntively "-D hadoop.pipes.java.recordreader=false".

However, when I ran with above configuration and with my record reader, I
got error
Hadoop Pipes Exception: RecordReader defined when not needed. at
impl/HadoopPipes.cc:798 in virtual void
HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)

It pipes did not seem to pick up my setting of
hadoop.pipes.java.recordreader as false.

I've tried using conf.xml or putting -D or the combine of both. None
worked. I've googled the whole day but could not find the reason. Did I
miss something here?

I am using hadoop-1.0.4.

Here is my conf.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>hadoop.pipes.executable</name>
    <value>bin/cpc</value>
  </property>
  <property>
    <name>hadoop.pipes.java.recordreader</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.pipes.java.recordwriter</name>
    <value>true</value>
  </property>
</configuration>

Here is the command

$HADOOP pipes \
-conf $CONF \
-files 0 \
-libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
-input $IN \
-output $OUT \
-program bin/$NAME \
-reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
-inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat

where $CONF is full path to conf.xml

I could provide more info if that hellps to determine the reason.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB