Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> WholeFileInputFormat  with  streaming


Copy link to this message
-
WholeFileInputFormat  with  streaming
Hi,

    I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat

    so, I download code from :  
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java

    and package the java file with :
package org.apache.hadoop.streaming;

solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java  to  hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java

there is a lot of error

solution B:
compile the java file   WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar   to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat .......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat

what's wrong with the two solution? or is there any new solution?

thx.

Ring
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB