Hi All,

Is there a way for me to set global counters in Mapper and access them from
reducer?
Could you suggest how I can acheve this?

Thanks
Gayatri

Hi

You can create your own counters like

enum CountFruits {
Apple,
Mango,
Banana
}
And in your mapper class when you see condition to increment , you can use
Reporter incrCounter method to do the same.

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html#incrCounter(java.lang.Enum,%20long)

e.g
// I saw Apple increment it by one
reporter.incrCounter(CountFruits.Apple,1);

Now you can access them using job.getCounters

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()

Hope this helps

Regards,

Jagat Singh
On Fri, Apr 20, 2012 at 9:43 PM, Gayatri Rao <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> Is there a way for me to set global counters in Mapper and access them from
> reducer?
> Could you suggest how I can acheve this?
>
> Thanks
> Gayatri
>

Yes U can use user defined counter as Jagat suggeted.

Counter can be enum as Jagat described or any string which are called dynamic counters.

It is easier to use Enum counter than dynamic counters, finally it depends on your use case :)

Amith
________________________________________
From: Jagat [[EMAIL PROTECTED]]
Sent: Saturday, April 21, 2012 12:25 AM
To: [EMAIL PROTECTED]
Subject: Re: Accessing global Counters

Hi

You can create your own counters like

enum CountFruits {
Apple,
Mango,
Banana
}
And in your mapper class when you see condition to increment , you can use
Reporter incrCounter method to do the same.

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html#incrCounter(java.lang.Enum,%20long)

e.g
// I saw Apple increment it by one
reporter.incrCounter(CountFruits.Apple,1);

Now you can access them using job.getCounters

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()

Hope this helps

Regards,

Jagat Singh
On Fri, Apr 20, 2012 at 9:43 PM, Gayatri Rao <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> Is there a way for me to set global counters in Mapper and access them from
> reducer?
> Could you suggest how I can acheve this?
>
> Thanks
> Gayatri
>

There was a discussion about this several months ago

http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201112.mbox/%[EMAIL PROTECTED]%3E

The conclusion is that if you want to read them from the reducer you are going to have to do something special until someone finds time to implement it as part of.

https://issues.apache.org/jira/browse/MAPREDUCE-3520

--Bobby Evans
On 4/20/12 11:36 AM, "Amith D K" <[EMAIL PROTECTED]> wrote:

Yes U can use user defined counter as Jagat suggeted.

Counter can be enum as Jagat described or any string which are called dynamic counters.

It is easier to use Enum counter than dynamic counters, finally it depends on your use case :)

Amith
________________________________________
From: Jagat [[EMAIL PROTECTED]]
Sent: Saturday, April 21, 2012 12:25 AM
To: [EMAIL PROTECTED]
Subject: Re: Accessing global Counters

Hi

You can create your own counters like

enum CountFruits {
Apple,
Mango,
Banana
}
And in your mapper class when you see condition to increment , you can use
Reporter incrCounter method to do the same.

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html#incrCounter(java.lang.Enum,%20long)

e.g
// I saw Apple increment it by one
reporter.incrCounter(CountFruits.Apple,1);

Now you can access them using job.getCounters

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()

Hope this helps

Regards,

Jagat Singh
On Fri, Apr 20, 2012 at 9:43 PM, Gayatri Rao <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> Is there a way for me to set global counters in Mapper and access them from
> reducer?
> Could you suggest how I can acheve this?
>
> Thanks
> Gayatri
>

No reducers can't access mapper counters.
---> maybe theres a way to intermediately put counters in the distributed cache???

Jay Vyas
MMSB
UCHC

On Apr 20, 2012, at 1:24 PM, Robert Evans <[EMAIL PROTECTED]> wrote:

> There was a discussion about this several months ago
>
> http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201112.mbox/%[EMAIL PROTECTED]%3E
>
> The conclusion is that if you want to read them from the reducer you are going to have to do something special until someone finds time to implement it as part of.
>
> https://issues.apache.org/jira/browse/MAPREDUCE-3520
>
> --Bobby Evans
>
>
> On 4/20/12 11:36 AM, "Amith D K" <[EMAIL PROTECTED]> wrote:
>
> Yes U can use user defined counter as Jagat suggeted.
>
> Counter can be enum as Jagat described or any string which are called dynamic counters.
>
> It is easier to use Enum counter than dynamic counters, finally it depends on your use case :)
>
> Amith
> ________________________________________
> From: Jagat [[EMAIL PROTECTED]]
> Sent: Saturday, April 21, 2012 12:25 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Accessing global Counters
>
> Hi
>
> You can create your own counters like
>
> enum CountFruits {
> Apple,
> Mango,
> Banana
> }
>
>
> And in your mapper class when you see condition to increment , you can use
> Reporter incrCounter method to do the same.
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html#incrCounter(java.lang.Enum,%20long)
>
> e.g
> // I saw Apple increment it by one
> reporter.incrCounter(CountFruits.Apple,1);
>
> Now you can access them using job.getCounters
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
>
> Hope this helps
>
> Regards,
>
> Jagat Singh
>
>
> On Fri, Apr 20, 2012 at 9:43 PM, Gayatri Rao <[EMAIL PROTECTED]> wrote:
>
>> Hi All,
>>
>> Is there a way for me to set global counters in Mapper and access them from
>> reducer?
>> Could you suggest how I can acheve this?
>>
>> Thanks
>> Gayatri
>>
>

Currently the DistributedCache is populated pre-Job run, hence both
Map and Reduce phases carry the same items. With MR2, the approach
Robert describes above should work better instead.

On Sat, Apr 21, 2012 at 5:21 AM, JAX <[EMAIL PROTECTED]> wrote:
> No reducers can't access mapper counters.
> ---> maybe theres a way to intermediately put counters in the distributed cache???
>
> Jay Vyas
> MMSB
> UCHC
>
> On Apr 20, 2012, at 1:24 PM, Robert Evans <[EMAIL PROTECTED]> wrote:
>
>> There was a discussion about this several months ago
>>
>> http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201112.mbox/%[EMAIL PROTECTED]%3E
>>
>> The conclusion is that if you want to read them from the reducer you are going to have to do something special until someone finds time to implement it as part of.
>>
>> https://issues.apache.org/jira/browse/MAPREDUCE-3520
>>
>> --Bobby Evans
>>
>>
>> On 4/20/12 11:36 AM, "Amith D K" <[EMAIL PROTECTED]> wrote:
>>
>> Yes U can use user defined counter as Jagat suggeted.
>>
>> Counter can be enum as Jagat described or any string which are called dynamic counters.
>>
>> It is easier to use Enum counter than dynamic counters, finally it depends on your use case :)
>>
>> Amith
>> ________________________________________
>> From: Jagat [[EMAIL PROTECTED]]
>> Sent: Saturday, April 21, 2012 12:25 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Accessing global Counters
>>
>> Hi
>>
>> You can create your own counters like
>>
>> enum CountFruits {
>> Apple,
>> Mango,
>> Banana
>> }
>>
>>
>> And in your mapper class when you see condition to increment , you can use
>> Reporter incrCounter method to do the same.
>>
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html#incrCounter(java.lang.Enum,%20long)
>>
>> e.g
>> // I saw Apple increment it by one
>> reporter.incrCounter(CountFruits.Apple,1);
>>
>> Now you can access them using job.getCounters
>>
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
>>
>> Hope this helps
>>
>> Regards,
>>
>> Jagat Singh
>>
>>
>> On Fri, Apr 20, 2012 at 9:43 PM, Gayatri Rao <[EMAIL PROTECTED]> wrote:
>>
>>> Hi All,
>>>
>>> Is there a way for me to set global counters in Mapper and access them from
>>> reducer?
>>> Could you suggest how I can acheve this?
>>>
>>> Thanks
>>> Gayatri
>>>
>>

--
Harsh J

Actually it's easier to use dynamic counters...

Sent from a remote device. Please excuse any typos...

Mike Segel

On Apr 20, 2012, at 11:36 AM, Amith D K <[EMAIL PROTECTED]> wrote:

> Yes U can use user defined counter as Jagat suggeted.
>
> Counter can be enum as Jagat described or any string which are called dynamic counters.
>
> It is easier to use Enum counter than dynamic counters, finally it depends on your use case :)
>
> Amith
> ________________________________________
> From: Jagat [[EMAIL PROTECTED]]
> Sent: Saturday, April 21, 2012 12:25 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Accessing global Counters
>
> Hi
>
> You can create your own counters like
>
> enum CountFruits {
> Apple,
> Mango,
> Banana
> }
>
>
> And in your mapper class when you see condition to increment , you can use
> Reporter incrCounter method to do the same.
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html#incrCounter(java.lang.Enum,%20long)
>
> e.g
> // I saw Apple increment it by one
> reporter.incrCounter(CountFruits.Apple,1);
>
> Now you can access them using job.getCounters
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
>
> Hope this helps
>
> Regards,
>
> Jagat Singh
>
>
> On Fri, Apr 20, 2012 at 9:43 PM, Gayatri Rao <[EMAIL PROTECTED]> wrote:
>
>> Hi All,
>>
>> Is there a way for me to set global counters in Mapper and access them from
>> reducer?
>> Could you suggest how I can acheve this?
>>
>> Thanks
>> Gayatri
>>
>