Hi,

I have around 4 jobs running in a controller.
How can i have a single unique counter present in all the jobs and
incremented where ever used in a job?

For example:Consider a counter ACount.
If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6.
Can i have the  counter displayed output in the jobtracker as
job1:2
job2:2
job3:7
job4:13

Thanks,
Subbu

Hi Subbu,

You're probably looking for something called "Distributed counters". Take a
look at this question at StackOverflow:
http://stackoverflow.com/questions/2671858/distributed-sequence-number-generation
Best regards,

Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E [EMAIL PROTECTED]

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

2012/8/28 Kasi Subrahmanyam <[EMAIL PROTECTED]>

> Hi,
>
> I have around 4 jobs running in a controller.
> How can i have a single unique counter present in all the jobs and
> incremented where ever used in a job?
>
> For example:Consider a counter ACount.
> If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6.
> Can i have the  counter displayed output in the jobtracker as
> job1:2
> job2:2
> job3:7
> job4:13
>
> Thanks,
> Subbu
>
>

Counters are per-job in Hadoop MapReduce. You need an external aggregator for such cross-job counters - for e.g. a node in Zookeeper.

Also, is it just for display or your job-logic depends on this? If it is the earlier, and if you don't have a problem with waiting till jobs finish, you can do a post-process on the counters of all jobs and calculate the aggregates.

Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Aug 28, 2012, at 1:20 AM, Kasi Subrahmanyam wrote:

> Hi,
>
> I have around 4 jobs running in a controller.
> How can i have a single unique counter present in all the jobs and incremented where ever used in a job?
>
> For example:Consider a counter ACount.
> If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6.
> Can i have the  counter displayed output in the jobtracker as
> job1:2
> job2:2
> job3:7
> job4:13
>
> Thanks,
> Subbu
>