Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Collecting MAP output in a Iterator


Copy link to this message
-
RE: Collecting MAP output in a Iterator
Siddharth Tiwari 2012-08-20, 21:46

Hi Bertrand,

You got me spot on,
I wanted to do a cross Join.
I have some data related to a particular type of payment card, we need to find all types of txn user did during the day. So in one file I have something linke below :-

USER MERCHANT ANMT TXN TS LAT LONG   -- actual big log

so we want a report want a report which shows how many events happened at what day.
so that way we can see how active was teh user in aparticular hour on a particular day

*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.”

"Maybe other people will try to limit me but I don't limit myself"
Date: Mon, 20 Aug 2012 22:28:50 +0200
Subject: Re: Collecting MAP output in a Iterator
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]

The question is why do you want to do that? How will you use the output?You really want a cross join? This a very expensive operation (in terms of memory).
Could you provide more context?

Regards
Bertrand

On Mon, Aug 20, 2012 at 10:24 PM, Lance Norskog <[EMAIL PROTECTED]> wrote:

The reducer has to open MAP2 explicitly. If it is a regular file, it

has to be in distributed cache.

On Mon, Aug 20, 2012 at 12:56 PM, Siddharth Tiwari

<[EMAIL PROTECTED]> wrote:

>

> Hi

>

> I have two mappers MAP1 and MAP2, which collect data from two different

> files,

> In reducer I want to traverse all keys and values of MAP2 for each key and

> value of  MAP1.

> How can I achieve it in one Reducer.

>

> Please suggest

>

>

> *------------------------*

> Cheers !!!

> Siddharth Tiwari

> Have a refreshing day !!!

> "Every duty is holy, and devotion to duty is the highest form of worship of

> God.”

> "Maybe other people will try to limit me but I don't limit myself"

--

Lance Norskog

[EMAIL PROTECTED]

--
Bertrand Dechoux