Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: custom writablecomparable with complex fields


Copy link to this message
-
Re: custom writablecomparable with complex fields
Harsh J 2013-08-31, 20:52
The idea behind write(…) and readFields(…) is simply that of ordering.
You need to write your custom objects (i.e. a representation of them)
in order, and read them back in the same order.

An example way of serializing a list would be to first serialize the
length (so you know how many you'll be needed to read back), and then
serialize each item appropriately, using delimiters, or using
length-prefixes just like lists.

Mainly, you're required to tackle the serialization/deserialization on your own.

This is one of the reasons I highly recommend using a library like
Apache Avro instead. Its more powerful, faster, and yet simple to use:
http://avro.apache.org/docs/current/gettingstartedjava.html and
http://avro.apache.org/docs/current/mr.html. It is also popular and
carries first-grade support on several other hadoop-ecosystem
projects, such as Flume and Crunch.

On Sun, Sep 1, 2013 at 1:23 AM, Adeel Qureshi <[EMAIL PROTECTED]> wrote:
> I want to write a custom writablecomparable object with two List objects
> within it ..
>
> public class CompositeKey implements WritableComparable {
>
> private List<JsonKey> groupBy;
> private List<JsonKey> sortBy;
> ...
> }
>
> what I am not sure about is how to write
>
> readFields and write methods for this object. Any help would be appreciated.
>
> Thanks
> Adeel

--
Harsh J