Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to java.lang.String


Copy link to this message
-
Re: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to java.lang.String
Hi,

              To make it simple what Harsh said is,

              MapReduce framework only works on serializable (i.e.,
writable) objects. In case of keys it should be serializable and comparable.
              So you cannot use the basic datatypes like int, String, long
etc rather you should use the similar types in Hadoop like IntWritable,
Text, LongWritable etc.

              The above error you got because the sourcekey and sourcevalue
are determined by the type of inputformat you use.
              I believe you are using TextInputFormat in your job so the
key will be Object/LongWritable and value will be Text,
              so when the framework is trying to pass those LongWritable to
your mapper it is throwing the classcast exception at runtime.

Best,
Mahesh Balija,
Calsoft Labs.

On Tue, Nov 20, 2012 at 12:41 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi,
>
> 1. Map/Reduce in 1.x. does not know how to efficiently and
> automatically serialize regular Java types such as String, Long, etc..
> There is experimental (and I wouldn't recommend using it either) Java
> serialization support for such types in 2.x releases if you enable the
> JavaSerialization classes. Please use Writable-based native type
> classes (such as Text for String, LongWritable for long, etc.).
>
> 2. The type checking is done based on configuration at runtime. What
> you set as JobConf.setOutput[Key/Value]Class and
> JobConf.setMapOutput[Key/Value]Class (or similar new API equivalents)
> will be checked for passing instances of key and value objects during
> runtime. By default, these are LongWritable (key) and Text (value) and
> you see the former in your error.
>
> On Mon, Nov 19, 2012 at 8:05 AM, Utester Utester <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > I have Hadoop 1.0.3 running on Ubuntu Linux. I am playing around with a
> > simple Map-Reduce job.
> >
> > Details of the code
> > ============> > 1. Since this exercise is mainly to learn basic Hadoop APIs and learn
> how to
> > run these jobs, the logic in the Map job is irrelevant.
> > 2. The code snippet is below with just the APIs (which I believe is
> causing
> > a problem - PLEASE correct me if I am wrong and I will post the entire
> code
> > snippet)
> > public
> > class JobOrganizer
> >
> > ....
> > ....
> > public
> > static class JobOrganizerMapper extends MapReduceBase implements
> > Mapper<String, String, String, String> {
> >
> > @Override
> > public void map(String sourceKey, String sourceValue,
> > OutputCollector<String, String> outputC,
> > Reporter reporter) throws IOException {
> > ...
> > ...
> > }
> >
> > }
> >
> > Exception:
> >
> > hduser@utester-VirtualBox:/usr/local/hadoop/bin$
> > /usr/local/hadoop/bin/hadoop jar
> >
> ~/HadoopCodeProjectsFolder/JobOrganizerMapRedProject/11182012/HadoopMapRedProject.jar
> > org.u.hadoopmapred.JobOrganizer
> > Warning: $HADOOP_HOME is deprecated.
> > 12/11/18 16:36:22 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the same.
> > 12/11/18 16:36:22 INFO util.NativeCodeLoader: Loaded the native-hadoop
> > library
> > 12/11/18 16:36:22 WARN snappy.LoadSnappy: Snappy native library not
> loaded
> > 12/11/18 16:36:22 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 1
> > 12/11/18 16:36:22 INFO mapred.JobClient: Running job:
> job_201211181608_0002
> > 12/11/18 16:36:23 INFO mapred.JobClient:  map 0% reduce 0%
> > 12/11/18 16:36:42 INFO mapred.JobClient: Task Id :
> > attempt_201211181608_0002_m_000000_0, Status : FAILED
> > java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
> > cast to java.lang.String
> >  at
> >
> org.u.hadoopmapred.JobOrganizer$JobOrganizerMapper.map(JobOrganizer.java:1)
> >  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> >
> >
> > The Mapper Interface as per the javadoc is: Interface Mapper <K1, V1, K2,
> > V2> and its map function is: map(K1 key, V1 value, OutputCollector<K2,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB