Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Pipes Property Not Passed In


+
Xun TANG 2013-04-24, 05:14
Copy link to this message
-
Re: Pipes Property Not Passed In
I think I pinned down the problem, but still can't find a solution.

It seems like 'hadoop.pipes.java.recordreader' is not the correct property
name to use?
Because I tried setting other properties (e.g. hadoop.pipes.executable) in
conf.xml and they were effectively passed in to pipes when the program
runs.

Where could I find the source code of pipes reading conf.xml file? Where
could I find a list of all properties (for pipes) and their default values?
Best I could find is this
http://hadoop.apache.org/docs/r1.1.2/commands_manual.html
but it does not list all properties.

Any suggestion?

Thanks,
Alice
On Tue, Apr 23, 2013 at 10:14 PM, Xun TANG <[EMAIL PROTECTED]> wrote:

> I implemented my own InputFormat/RecordReader, and I try to run it with
> Hadoop Pipes. I understand I could pass in properties to Pipes program by
> either:
>
> <property>
>         <name>hadoop.pipes.java.recordreader</name>
>         <value>false</value>
> </property>
>
> or alterntively "-D hadoop.pipes.java.recordreader=false".
>
> However, when I ran with above configuration and with my record reader, I
> got error
> Hadoop Pipes Exception: RecordReader defined when not needed. at
> impl/HadoopPipes.cc:798 in virtual void
> HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)
>
> It pipes did not seem to pick up my setting of
> hadoop.pipes.java.recordreader as false.
>
> I've tried using conf.xml or putting -D or the combine of both. None
> worked. I've googled the whole day but could not find the reason. Did I
> miss something here?
>
> I am using hadoop-1.0.4.
>
> Here is my conf.xml
>
> <?xml version="1.0"?>
> <configuration>
>   <property>
>      <name>hadoop.pipes.executable</name>
>     <value>bin/cpc</value>
>   </property>
>   <property>
>     <name>hadoop.pipes.java.recordreader</name>
>     <value>false</value>
>   </property>
>   <property>
>     <name>hadoop.pipes.java.recordwriter</name>
>     <value>true</value>
>   </property>
> </configuration>
>
> Here is the command
>
> $HADOOP pipes \
> -conf $CONF \
> -files 0 \
> -libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
> -input $IN \
> -output $OUT \
> -program bin/$NAME \
> -reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
> -inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat
>
> where $CONF is full path to conf.xml
>
> I could provide more info if that hellps to determine the reason.
>
>