Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Pipes Property Not Passed In


+
Xun TANG 2013-04-24, 05:14
Copy link to this message
-
Re: Pipes Property Not Passed In
I think I pinned down the problem, but still can't find a solution.

It seems like 'hadoop.pipes.java.recordreader' is not the correct property
name to use?
Because I tried setting other properties (e.g. hadoop.pipes.executable) in
conf.xml and they were effectively passed in to pipes when the program
runs.

Where could I find the source code of pipes reading conf.xml file? Where
could I find a list of all properties (for pipes) and their default values?
Best I could find is this
http://hadoop.apache.org/docs/r1.1.2/commands_manual.html
but it does not list all properties.

Any suggestion?

Thanks,
Alice
On Tue, Apr 23, 2013 at 10:14 PM, Xun TANG <[EMAIL PROTECTED]> wrote:

> I implemented my own InputFormat/RecordReader, and I try to run it with
> Hadoop Pipes. I understand I could pass in properties to Pipes program by
> either:
>
> <property>
>         <name>hadoop.pipes.java.recordreader</name>
>         <value>false</value>
> </property>
>
> or alterntively "-D hadoop.pipes.java.recordreader=false".
>
> However, when I ran with above configuration and with my record reader, I
> got error
> Hadoop Pipes Exception: RecordReader defined when not needed. at
> impl/HadoopPipes.cc:798 in virtual void
> HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)
>
> It pipes did not seem to pick up my setting of
> hadoop.pipes.java.recordreader as false.
>
> I've tried using conf.xml or putting -D or the combine of both. None
> worked. I've googled the whole day but could not find the reason. Did I
> miss something here?
>
> I am using hadoop-1.0.4.
>
> Here is my conf.xml
>
> <?xml version="1.0"?>
> <configuration>
>   <property>
>      <name>hadoop.pipes.executable</name>
>     <value>bin/cpc</value>
>   </property>
>   <property>
>     <name>hadoop.pipes.java.recordreader</name>
>     <value>false</value>
>   </property>
>   <property>
>     <name>hadoop.pipes.java.recordwriter</name>
>     <value>true</value>
>   </property>
> </configuration>
>
> Here is the command
>
> $HADOOP pipes \
> -conf $CONF \
> -files 0 \
> -libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
> -input $IN \
> -output $OUT \
> -program bin/$NAME \
> -reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
> -inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat
>
> where $CONF is full path to conf.xml
>
> I could provide more info if that hellps to determine the reason.
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB