Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - ArrayWritable usage


Copy link to this message
-
Re: ArrayWritable usage
Joey Echeverria 2011-07-04, 19:00
ArrayWritable doesn't serialize type information. You need to subclass it
(e.g. IntArrayWritable) and create a no arg constructor which calls
super(IntWritable.class).

Use this instead of ArrayWritable directly. If you want to store more than
one type, look at the source for MapWritable to see how it generates type
codes.

-Joey
On Jul 4, 2011 2:55 PM, "Dhruv Kumar" <[EMAIL PROTECTED]> wrote:
> I'm having some difficulty with using ArrayWritable in the following test
> code:
>
> ArrayWritable array = new ArrayWritable(IntWritable.class);
>
> IntWritable[] ints = new IntWritable[4];
>
> for (int i =0 ; i < 4; i++) {
> ints[i] = new IntWritable(i);
>
> }
>
> array.set(ints);
>
> writer.append(new LongWritable(1), array);
>
> writer.close();
>
> After starting the map reduce application's test, I'm getting the
following
> errors which I suspect are due to improper use of ArrayWritable. Can
someone
> tell me if the code given above is correct?
>
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.hadoop.io.ArrayWritable.<init>()
> at
>
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> at
>
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
> at
>
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> at
>
org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
> at
>
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
> at
>
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
> at
>
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.io.ArrayWritable.<init>()
> at java.lang.Class.getConstructor0(Class.java:2706)
> at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> at
>
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> ... 15 more