Hi

I have two mappers MAP1 and MAP2, which collect data from two different files,
In reducer I want to traverse all keys and values of MAP2 for each key and value of  MAP1.
How can I achieve it in one Reducer.

Please suggest

*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.”

"Maybe other people will try to limit me but I don't limit myself"
     

The reducer has to open MAP2 explicitly. If it is a regular file, it
has to be in distributed cache.

On Mon, Aug 20, 2012 at 12:56 PM, Siddharth Tiwari
<[EMAIL PROTECTED]> wrote:
>
> Hi
>
> I have two mappers MAP1 and MAP2, which collect data from two different
> files,
> In reducer I want to traverse all keys and values of MAP2 for each key and
> value of  MAP1.
> How can I achieve it in one Reducer.
>
> Please suggest
>
>
> *------------------------*
> Cheers !!!
> Siddharth Tiwari
> Have a refreshing day !!!
> "Every duty is holy, and devotion to duty is the highest form of worship of
> God.”
> "Maybe other people will try to limit me but I don't limit myself"

--
Lance Norskog
[EMAIL PROTECTED]

The question is why do you want to do that? How will you use the output?
You really want a cross join? This a very expensive operation (in terms of
memory).

Could you provide more context?

Regards

Bertrand

On Mon, Aug 20, 2012 at 10:24 PM, Lance Norskog <[EMAIL PROTECTED]> wrote:

> The reducer has to open MAP2 explicitly. If it is a regular file, it
> has to be in distributed cache.
>
> On Mon, Aug 20, 2012 at 12:56 PM, Siddharth Tiwari
> <[EMAIL PROTECTED]> wrote:
> >
> > Hi
> >
> > I have two mappers MAP1 and MAP2, which collect data from two different
> > files,
> > In reducer I want to traverse all keys and values of MAP2 for each key
> and
> > value of  MAP1.
> > How can I achieve it in one Reducer.
> >
> > Please suggest
> >
> >
> > *------------------------*
> > Cheers !!!
> > Siddharth Tiwari
> > Have a refreshing day !!!
> > "Every duty is holy, and devotion to duty is the highest form of worship
> of
> > God.”
> > "Maybe other people will try to limit me but I don't limit myself"
>
>
>
> --
> Lance Norskog
> [EMAIL PROTECTED]
>

--
Bertrand Dechoux

Hi Bertrand,

You got me spot on,
I wanted to do a cross Join.
I have some data related to a particular type of payment card, we need to find all types of txn user did during the day. So in one file I have something linke below :-

USER MERCHANT ANMT TXN TS LAT LONG   -- actual big log

so we want a report want a report which shows how many events happened at what day.
so that way we can see how active was teh user in aparticular hour on a particular day

*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.”

"Maybe other people will try to limit me but I don't limit myself"
Date: Mon, 20 Aug 2012 22:28:50 +0200
Subject: Re: Collecting MAP output in a Iterator
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]

The question is why do you want to do that? How will you use the output?You really want a cross join? This a very expensive operation (in terms of memory).
Could you provide more context?

Regards
Bertrand

On Mon, Aug 20, 2012 at 10:24 PM, Lance Norskog <[EMAIL PROTECTED]> wrote:

The reducer has to open MAP2 explicitly. If it is a regular file, it

has to be in distributed cache.

On Mon, Aug 20, 2012 at 12:56 PM, Siddharth Tiwari

<[EMAIL PROTECTED]> wrote:

>

> Hi

>

> I have two mappers MAP1 and MAP2, which collect data from two different

> files,

> In reducer I want to traverse all keys and values of MAP2 for each key and

> value of  MAP1.

> How can I achieve it in one Reducer.

>

> Please suggest

>

>

> *------------------------*

> Cheers !!!

> Siddharth Tiwari

> Have a refreshing day !!!

> "Every duty is holy, and devotion to duty is the highest form of worship of

> God.”

> "Maybe other people will try to limit me but I don't limit myself"

--

Lance Norskog

[EMAIL PROTECTED]

--
Bertrand Dechoux

     

This article has a good example, which might help:

http://www.ibm.com/developerworks/java/library/j-javadev2-15/index.html

SS
On Aug 20, 2012, at 2:46 PM, Siddharth Tiwari wrote:

> Hi Bertrand,
>
> You got me spot on,
> I wanted to do a cross Join.
> I have some data related to a particular type of payment card, we need to find all types of txn user did during the day. So in one file I have something linke below :-
>
> USER MERCHANT ANMT TXN TS LAT LONG   -- actual big log
>
> so we want a report want a report which shows how many events happened at what day.
> so that way we can see how active was teh user in aparticular hour on a particular day
>
>
>
>
> *------------------------*
> Cheers !!!
> Siddharth Tiwari
> Have a refreshing day !!!
> "Every duty is holy, and devotion to duty is the highest form of worship of God.”
> "Maybe other people will try to limit me but I don't limit myself"
>
>
> Date: Mon, 20 Aug 2012 22:28:50 +0200
> Subject: Re: Collecting MAP output in a Iterator
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> The question is why do you want to do that? How will you use the output?
> You really want a cross join? This a very expensive operation (in terms of memory).
>
> Could you provide more context?
>
> Regards
>
> Bertrand
>
> On Mon, Aug 20, 2012 at 10:24 PM, Lance Norskog <[EMAIL PROTECTED]> wrote:
> The reducer has to open MAP2 explicitly. If it is a regular file, it
> has to be in distributed cache.
>
> On Mon, Aug 20, 2012 at 12:56 PM, Siddharth Tiwari
> <[EMAIL PROTECTED]> wrote:
> >
> > Hi
> >
> > I have two mappers MAP1 and MAP2, which collect data from two different
> > files,
> > In reducer I want to traverse all keys and values of MAP2 for each key and
> > value of  MAP1.
> > How can I achieve it in one Reducer.
> >
> > Please suggest
> >
> >
> > *------------------------*
> > Cheers !!!
> > Siddharth Tiwari
> > Have a refreshing day !!!
> > "Every duty is holy, and devotion to duty is the highest form of worship of
> > God.”
> > "Maybe other people will try to limit me but I don't limit myself"
>
>
>
> --
> Lance Norskog
> [EMAIL PROTECTED]
>
>
>
> --
> Bertrand Dechoux

Hi Siddharth

Cross joins are implemented in a well optimized manner in Hive and Pig.
Your use case is a good fit for Hive or pig , it'd be better going for
those rather than spending much time on implementing the MR jobs yourself.

Regards
Bejoy KS

Hi Siddharth

To add on if one of your data sets/tables are small enough to be fit in
memory, you can distributed them in over Distribute cache and use it as a
look up while streaming the larger data to perform the join. With this you
can totally avoid a reduce phase in your join there by giving
a performance edge to your jobs.

Details on map join in hive :
https://cwiki.apache.org/Hive/joinoptimization.html
http://hive.apache.org/docs/r0.9.0/language_manual/joins.html

Regards
Bejoy KS