To add on, have a look at hive and pig. Those are perfect fit for similar use cases.

Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: "Bejoy KS" <[EMAIL PROTECTED]>
Date: Tue, 12 Jun 2012 13:04:33
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Need logical help

Hi Girish

You can achice this using reduce side joins. Use MultipleInputFormat for parsing two different sets of log files.

 
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Girish Ravi <[EMAIL PROTECTED]>
Date: Tue, 12 Jun 2012 12:59:32
To: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Need logical help

Hi All,

I am thinking of a condition where the data in two log files are to be compared, can I use Map-Reduce to do this?

I have one log file (LOG1) which has user ID and dept ID and another log file (LOG2) has some rows which has user ID and dept ID and other data.
Can I compare the data where LOG1.userID = LOG2.userID and LOG1.deptID = LOG2.deptID?

If so any suggestion to implement the mapper for this?

Regards
Girish
Ph: +91-9916212114

Hi All,

I am thinking of a condition where the data in two log files are to be compared, can I use Map-Reduce to do this?

I have one log file (LOG1) which has user ID and dept ID and another log file (LOG2) has some rows which has user ID and dept ID and other data.
Can I compare the data where LOG1.userID = LOG2.userID and LOG1.deptID = LOG2.deptID?

If so any suggestion to implement the mapper for this?

Regards
Girish
Ph: +91-9916212114

Hello Girish,

This is nothing but lookup join usecase.

Just do a quick search on joins using mapreduce.

Regards,

Jagat Singh

-----------
Sent from Mobile , short and crisp.
On 12-Jun-2012 6:31 PM, "Girish Ravi" <[EMAIL PROTECTED]> wrote:

>  Hi All,
>
> I am thinking of a condition where the data in two log files are to be
> compared, can I use Map-Reduce to do this?
>
> I have one log file (LOG1) which has user ID and dept ID and another log
> file (LOG2) has some rows which has user ID and dept ID and other data.
> Can I compare the data where LOG1.userID = LOG2.userID and LOG1.deptID > LOG2.deptID?
>
> If so any suggestion to implement the mapper for this?
>
> Regards
> Girish
> Ph: +91-9916212114
>

Hi Girish

You can achice this using reduce side joins. Use MultipleInputFormat for parsing two different sets of log files.

 
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Girish Ravi <[EMAIL PROTECTED]>
Date: Tue, 12 Jun 2012 12:59:32
To: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Need logical help

Hi All,

I am thinking of a condition where the data in two log files are to be compared, can I use Map-Reduce to do this?

I have one log file (LOG1) which has user ID and dept ID and another log file (LOG2) has some rows which has user ID and dept ID and other data.
Can I compare the data where LOG1.userID = LOG2.userID and LOG1.deptID = LOG2.deptID?

If so any suggestion to implement the mapper for this?

Regards
Girish
Ph: +91-9916212114