Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: WholeFileInputFormat with streaming


Copy link to this message
-
Re: WholeFileInputFormat with streaming
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?

2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:

> Hi,
>
>     I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
>     so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
>     and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file   WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar   to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB