Hi,

I have Hadoop 1.0.3 running on Ubuntu Linux. I am playing around with a simple Map-Reduce job.

Details of the code
============1. Since this exercise is mainly to learn basic Hadoop APIs and learn how to run these jobs, the logic in the Map job is irrelevant.
2. The code snippet is below with just the APIs (which I believe is causing a problem - PLEASE correct me if I am wrong and I will post the entire code snippet)public
 
....
....public
  Reporter reporter)
...
...
}publicvoidmap(String sourceKey, String sourceValue, OutputCollector<String, String> outputC,throwsIOException {
 
}
 
Exception:

hduser@utester-VirtualBox:/usr/local/hadoop/bin$ /usr/local/hadoop/bin/hadoop jar ~/HadoopCodeProjectsFolder/JobOrganizerMapRedProject/11182012/HadoopMapRedProject.jar org.u.hadoopmapred.JobOrganizer
Warning: $HADOOP_HOME is deprecated.
12/11/18 16:36:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/11/18 16:36:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/11/18 16:36:22 WARN snappy.LoadSnappy: Snappy native library not loaded
12/11/18 16:36:22 INFO mapred.FileInputFormat: Total input paths to process : 1
12/11/18 16:36:22 INFO mapred.JobClient: Running job: job_201211181608_0002
12/11/18 16:36:23 INFO mapred.JobClient:  map 0% reduce 0%
12/11/18 16:36:42 INFO mapred.JobClient: Task Id : attempt_201211181608_0002_m_000000_0, Status : FAILED
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to java.lang.String
 at org.u.hadoopmapred.JobOrganizer$JobOrganizerMapper.map(JobOrganizer.java:1)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 
The Mapper Interface as per the javadoc is: Interface Mapper <K1, V1, K2, V2> and its map function is: map(K1 key, V1 value, OutputCollector<K2, V2> output, Reporter). I wanted to parameterize K1, V1, K2, V2 to all be String. Is something wrong in the way I am thinking? Is this what is wrong? I have found similar questions on the internet but the asnwers did not clarify how I am breaking the Mapper contract (I did not have any compile errors - just a runtime error).
 
 
 
Thanks 
@OverridestaticclassJobOrganizerMapper extendsMapReduceBase implementsMapper<String, String, String, String> {classJobOrganizer

Hi,

1. Map/Reduce in 1.x. does not know how to efficiently and
automatically serialize regular Java types such as String, Long, etc..
There is experimental (and I wouldn't recommend using it either) Java
serialization support for such types in 2.x releases if you enable the
JavaSerialization classes. Please use Writable-based native type
classes (such as Text for String, LongWritable for long, etc.).

2. The type checking is done based on configuration at runtime. What
you set as JobConf.setOutput[Key/Value]Class and
JobConf.setMapOutput[Key/Value]Class (or similar new API equivalents)
will be checked for passing instances of key and value objects during
runtime. By default, these are LongWritable (key) and Text (value) and
you see the former in your error.

On Mon, Nov 19, 2012 at 8:05 AM, Utester Utester <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have Hadoop 1.0.3 running on Ubuntu Linux. I am playing around with a
> simple Map-Reduce job.
>
> Details of the code
> ============> 1. Since this exercise is mainly to learn basic Hadoop APIs and learn how to
> run these jobs, the logic in the Map job is irrelevant.
> 2. The code snippet is below with just the APIs (which I believe is causing
> a problem - PLEASE correct me if I am wrong and I will post the entire code
> snippet)
> public
> class JobOrganizer
>
> ....
> ....
> public
> static class JobOrganizerMapper extends MapReduceBase implements
> Mapper<String, String, String, String> {
>
> @Override
> public void map(String sourceKey, String sourceValue,
> OutputCollector<String, String> outputC,
> Reporter reporter) throws IOException {
> ...
> ...
> }
>
> }
>
> Exception:
>
> hduser@utester-VirtualBox:/usr/local/hadoop/bin$
> /usr/local/hadoop/bin/hadoop jar
> ~/HadoopCodeProjectsFolder/JobOrganizerMapRedProject/11182012/HadoopMapRedProject.jar
> org.u.hadoopmapred.JobOrganizer
> Warning: $HADOOP_HOME is deprecated.
> 12/11/18 16:36:22 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12/11/18 16:36:22 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 12/11/18 16:36:22 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/11/18 16:36:22 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 12/11/18 16:36:22 INFO mapred.JobClient: Running job: job_201211181608_0002
> 12/11/18 16:36:23 INFO mapred.JobClient:  map 0% reduce 0%
> 12/11/18 16:36:42 INFO mapred.JobClient: Task Id :
> attempt_201211181608_0002_m_000000_0, Status : FAILED
> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
> cast to java.lang.String
>  at
> org.u.hadoopmapred.JobOrganizer$JobOrganizerMapper.map(JobOrganizer.java:1)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>
>
> The Mapper Interface as per the javadoc is: Interface Mapper <K1, V1, K2,
> V2> and its map function is: map(K1 key, V1 value, OutputCollector<K2, V2>
> output, Reporter). I wanted to parameterize K1, V1, K2, V2 to all be String.
> Is something wrong in the way I am thinking? Is this what is wrong? I have
> found similar questions on the internet but the asnwers did not clarify how
> I am breaking the Mapper contract (I did not have any compile errors - just
> a runtime error).
>
>
>
> Thanks

--
Harsh J

Hi,

              To make it simple what Harsh said is,

              MapReduce framework only works on serializable (i.e.,
writable) objects. In case of keys it should be serializable and comparable.
              So you cannot use the basic datatypes like int, String, long
etc rather you should use the similar types in Hadoop like IntWritable,
Text, LongWritable etc.

              The above error you got because the sourcekey and sourcevalue
are determined by the type of inputformat you use.
              I believe you are using TextInputFormat in your job so the
key will be Object/LongWritable and value will be Text,
              so when the framework is trying to pass those LongWritable to
your mapper it is throwing the classcast exception at runtime.

Best,
Mahesh Balija,
Calsoft Labs.

On Tue, Nov 20, 2012 at 12:41 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi,
>
> 1. Map/Reduce in 1.x. does not know how to efficiently and
> automatically serialize regular Java types such as String, Long, etc..
> There is experimental (and I wouldn't recommend using it either) Java
> serialization support for such types in 2.x releases if you enable the
> JavaSerialization classes. Please use Writable-based native type
> classes (such as Text for String, LongWritable for long, etc.).
>
> 2. The type checking is done based on configuration at runtime. What
> you set as JobConf.setOutput[Key/Value]Class and
> JobConf.setMapOutput[Key/Value]Class (or similar new API equivalents)
> will be checked for passing instances of key and value objects during
> runtime. By default, these are LongWritable (key) and Text (value) and
> you see the former in your error.
>
> On Mon, Nov 19, 2012 at 8:05 AM, Utester Utester <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > I have Hadoop 1.0.3 running on Ubuntu Linux. I am playing around with a
> > simple Map-Reduce job.
> >
> > Details of the code
> > ============> > 1. Since this exercise is mainly to learn basic Hadoop APIs and learn
> how to
> > run these jobs, the logic in the Map job is irrelevant.
> > 2. The code snippet is below with just the APIs (which I believe is
> causing
> > a problem - PLEASE correct me if I am wrong and I will post the entire
> code
> > snippet)
> > public
> > class JobOrganizer
> >
> > ....
> > ....
> > public
> > static class JobOrganizerMapper extends MapReduceBase implements
> > Mapper<String, String, String, String> {
> >
> > @Override
> > public void map(String sourceKey, String sourceValue,
> > OutputCollector<String, String> outputC,
> > Reporter reporter) throws IOException {
> > ...
> > ...
> > }
> >
> > }
> >
> > Exception:
> >
> > hduser@utester-VirtualBox:/usr/local/hadoop/bin$
> > /usr/local/hadoop/bin/hadoop jar
> >
> ~/HadoopCodeProjectsFolder/JobOrganizerMapRedProject/11182012/HadoopMapRedProject.jar
> > org.u.hadoopmapred.JobOrganizer
> > Warning: $HADOOP_HOME is deprecated.
> > 12/11/18 16:36:22 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the same.
> > 12/11/18 16:36:22 INFO util.NativeCodeLoader: Loaded the native-hadoop
> > library
> > 12/11/18 16:36:22 WARN snappy.LoadSnappy: Snappy native library not
> loaded
> > 12/11/18 16:36:22 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 1
> > 12/11/18 16:36:22 INFO mapred.JobClient: Running job:
> job_201211181608_0002
> > 12/11/18 16:36:23 INFO mapred.JobClient:  map 0% reduce 0%
> > 12/11/18 16:36:42 INFO mapred.JobClient: Task Id :
> > attempt_201211181608_0002_m_000000_0, Status : FAILED
> > java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
> > cast to java.lang.String
> >  at
> >
> org.u.hadoopmapred.JobOrganizer$JobOrganizerMapper.map(JobOrganizer.java:1)
> >  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> >
> >
> > The Mapper Interface as per the javadoc is: Interface Mapper <K1, V1, K2,
> > V2> and its map function is: map(K1 key, V1 value, OutputCollector<K2,