Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Joining Avro input files in using Java mapreduce


Copy link to this message
-
Re: Joining Avro input files in using Java mapreduce
Sripad Sriram 2013-04-24, 19:02
Hey Martin,

I think those classes refer to outputting to multiple files rather than
reading from multiple files, which is what's needed for a reduce-side join.

thanks,
Sripad
On Wed, Apr 24, 2013 at 3:35 AM, Martin Kleppmann <[EMAIL PROTECTED]>wrote:

> Hey Sripad,
>
> Take a look at AvroMultipleInputs.
>
> http://avro.apache.org/docs/1.7.4/api/java/org/apache/avro/mapred/AvroMultipleOutputs.html(mapred version)
>
> http://avro.apache.org/docs/1.7.4/api/java/org/apache/avro/mapreduce/AvroMultipleOutputs.html(mapreduce version)
>
> Martin
>
>
> On 23 April 2013 17:01, Sripad Sriram <[EMAIL PROTECTED]> wrote:
>
>> Hey folks,
>>
>> Aware that I can use Pig, Hive, etc to join avro files together, but I
>> have several use cases where I need to perform a reduce-side join on two
>> avro files. MultipleInputs doesn't seem to like AvroInputFormat - any
>> thoughts?
>>
>> thanks!
>> Sripad
>>
>
>