Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Re: Hadoop Avro Question


+
Harsh J 2013-05-01, 05:06
Copy link to this message
-
Re: Hadoop Avro Question
For the TOP, check out
https://issues.apache.org/jira/browse/MAPREDUCE-4574 which we fixed in
Hadoop recently to allow full reuse with Avro.

On Wed, May 1, 2013 at 10:49 AM, Rahul Bhattacharjee
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> Looks like a lot of other classes are also to be rewritten for avro. Like
> the total sort partitioner , which I think currently assumes writable as the
> io mechanism.
>
> I faced problem using with avro , so though of writing to the forum.
>
> Thanks a lot
> Rahul!
>
>
> On Wed, May 1, 2013 at 10:35 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Moving the question to Apache Avro's user@ lists. Please use the right
>> lists for the most relevant answers.
>>
>> Avro is a different serialization technique that intends to replace
>> the Writable serialization defaults in Hadoop. MR accepts a list of
>> serializers it can use for its key/value structures and isn't limited
>> to Writable in any way. Look up the property "io.serializations" in
>> your Hadoop's core-default.xml for more information.
>>
>> The Avro project also offers fast comparator classes that are used for
>> comparing the bytes/structures of Avro objects. This is mostly
>> auto-set for you when you use the MR framework as described at
>>
>> http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
>> (via AvroJob helper class).
>>
>> On Tue, Apr 30, 2013 at 6:51 PM, Rahul Bhattacharjee
>> <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > When dealing with Avro data files in MR jobs ,we use AvroMapper , I
>> > noticed
>> > that the output of K and V of AvroMapper isnt writable and neither the
>> > key
>> > is comparable (these are AvroKey and AvroValue). As the general
>> > serialization mechanism is writable , how is the K,V pairs in case of
>> > avro ,
>> > travel across nodes?
>> >
>> > Thanks,
>> > Rahul
>>
>>
>>
>> --
>> Harsh J
>
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB