Lin Ma
2012-10-19, 16:09
Harsh J
2012-10-19, 16:19
Lin Ma
2012-10-19, 16:33
Harsh J
2012-10-19, 16:36
Jay Vyas
2012-10-19, 16:02
Lin Ma
2012-10-19, 10:50
Michael Segel
2012-10-19, 14:35
Lin Ma
2012-10-19, 16:27
Michael Segel
2012-10-19, 16:42
Lin Ma
2012-10-19, 16:56
Michael Segel
2012-10-19, 21:05
Lin Ma
2012-10-20, 03:27
Michael Segel
2012-10-20, 07:12
Lin Ma
2012-10-21, 06:45
Michael Segel
2012-10-21, 12:15
Lin Ma
2012-10-22, 08:00
Bertrand Dechoux
2012-10-19, 14:50
Jay Vyas
2012-10-19, 15:18
Bejoy KS
2012-10-19, 15:21
Lin Ma
2012-10-23, 07:12
-
Re: Hadoop counterLin Ma 2012-10-19, 16:09
Hi Harsh,
Thanks for the great reply. Two basic questions,
- Where the counters' value are stored for successful job? On JT?
- Supposing a specific job A completed successfully and updated related
counters, is it possible for another specific job B to read counters
updated by previous job A? If yes, how?
regards,
Lin
On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Bejoy is almost right, except that counters are reported upon progress
> of tasks itself (via TT heartbeats to JT actually), but the final
> counter representation is computed only with successful task reports
> the job received, not from any failed or killed ones.
>
> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:
> > Hi Jay
> >
> > Counters are reported at the end of a task to JT. So if a task fails the
> > counters from that task are not send to JT and hence won't be included in
> > the final value of counters from that Job.
> > Regards
> > Bejoy KS
> >
> > Sent from handheld, please excuse typos.
> > ________________________________
> > From: Jay Vyas <[EMAIL PROTECTED]>
> > Date: Fri, 19 Oct 2012 10:18:42 -0500
> > To: <[EMAIL PROTECTED]>
> > ReplyTo: [EMAIL PROTECTED]
> > Subject: Re: Hadoop counter
> >
> > Ah this answers alot about why some of my dynamic counters never show up
> and
> > i have to bite my nails waiting to see whats going on until the end of
> the
> > job- thanks.
> >
> > Another question: what happens if a task fails ? What happen to the
> > counters for it ? Do they dissappear into the ether? Or do they get
> merged
> > in with the counters from other tasks?
> >
> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>
> > wrote:
> >>
> >> And by default the number of counters is limited to 120 with the
> >> mapreduce.job.counters.limit property.
> >> They are useful for displaying short statistics about a job but should
> not
> >> be used for results (imho).
> >> I know people may misuse them but I haven't tried so I wouldn't be able
> to
> >> list the caveats.
> >>
> >> Regards
> >>
> >> Bertrand
> >>
> >>
> >> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <
> [EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>> As I understand it... each Task has its own counters and are
> >>> independently updated. As they report back to the JT, they update the
> >>> counter(s)' status.
> >>> The JT then will aggregate them.
> >>>
> >>> In terms of performance, Counters take up some memory in the JT so
> while
> >>> its OK to use them, if you abuse them, you can run in to issues.
> >>> As to limits... I guess that will depend on the amount of memory on the
> >>> JT machine, the size of the cluster (Number of TT) and the number of
> >>> counters.
> >>>
> >>> In terms of global accessibility... Maybe.
> >>>
> >>> The reason I say maybe is that I'm not sure by what you mean by
> globally
> >>> accessible.
> >>> If a task creates and implements a dynamic counter... I know that it
> will
> >>> eventually be reflected in the JT. However, I do not believe that a
> separate
> >>> Task could connect with the JT and see if the counter exists or if it
> could
> >>> get a value or even an accurate value since the updates are
> asynchronous.
> >>> Not to mention that I don't believe that the counters are aggregated
> until
> >>> the job ends. It would make sense that the JT maintains a unique
> counter for
> >>> each task until the tasks complete. (If a task fails, it would have to
> >>> delete the counters so that when the task is restarted the correct
> count is
> >>> maintained. ) Note, I haven't looked at the source code so I am
> probably
> >>> wrong.
> >>>
> >>> HTH
> >>> Mike
> >>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> >>>
> >>> Hi guys,
> >>>
> >>> I have some quick questions regarding to Hadoop counter,
> >>>
> >>> Hadoop counter (customer defined) is global accessible (for both read
> and
> >>> write) for all Mappers and Reducers in a job?
> >>> What is the performance and best practices of using Hadoop counters? I
Thanks for the great reply. Two basic questions,
- Where the counters' value are stored for successful job? On JT?
- Supposing a specific job A completed successfully and updated related
counters, is it possible for another specific job B to read counters
updated by previous job A? If yes, how?
regards,
Lin
On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Bejoy is almost right, except that counters are reported upon progress
> of tasks itself (via TT heartbeats to JT actually), but the final
> counter representation is computed only with successful task reports
> the job received, not from any failed or killed ones.
>
> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:
> > Hi Jay
> >
> > Counters are reported at the end of a task to JT. So if a task fails the
> > counters from that task are not send to JT and hence won't be included in
> > the final value of counters from that Job.
> > Regards
> > Bejoy KS
> >
> > Sent from handheld, please excuse typos.
> > ________________________________
> > From: Jay Vyas <[EMAIL PROTECTED]>
> > Date: Fri, 19 Oct 2012 10:18:42 -0500
> > To: <[EMAIL PROTECTED]>
> > ReplyTo: [EMAIL PROTECTED]
> > Subject: Re: Hadoop counter
> >
> > Ah this answers alot about why some of my dynamic counters never show up
> and
> > i have to bite my nails waiting to see whats going on until the end of
> the
> > job- thanks.
> >
> > Another question: what happens if a task fails ? What happen to the
> > counters for it ? Do they dissappear into the ether? Or do they get
> merged
> > in with the counters from other tasks?
> >
> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>
> > wrote:
> >>
> >> And by default the number of counters is limited to 120 with the
> >> mapreduce.job.counters.limit property.
> >> They are useful for displaying short statistics about a job but should
> not
> >> be used for results (imho).
> >> I know people may misuse them but I haven't tried so I wouldn't be able
> to
> >> list the caveats.
> >>
> >> Regards
> >>
> >> Bertrand
> >>
> >>
> >> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <
> [EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>> As I understand it... each Task has its own counters and are
> >>> independently updated. As they report back to the JT, they update the
> >>> counter(s)' status.
> >>> The JT then will aggregate them.
> >>>
> >>> In terms of performance, Counters take up some memory in the JT so
> while
> >>> its OK to use them, if you abuse them, you can run in to issues.
> >>> As to limits... I guess that will depend on the amount of memory on the
> >>> JT machine, the size of the cluster (Number of TT) and the number of
> >>> counters.
> >>>
> >>> In terms of global accessibility... Maybe.
> >>>
> >>> The reason I say maybe is that I'm not sure by what you mean by
> globally
> >>> accessible.
> >>> If a task creates and implements a dynamic counter... I know that it
> will
> >>> eventually be reflected in the JT. However, I do not believe that a
> separate
> >>> Task could connect with the JT and see if the counter exists or if it
> could
> >>> get a value or even an accurate value since the updates are
> asynchronous.
> >>> Not to mention that I don't believe that the counters are aggregated
> until
> >>> the job ends. It would make sense that the JT maintains a unique
> counter for
> >>> each task until the tasks complete. (If a task fails, it would have to
> >>> delete the counters so that when the task is restarted the correct
> count is
> >>> maintained. ) Note, I haven't looked at the source code so I am
> probably
> >>> wrong.
> >>>
> >>> HTH
> >>> Mike
> >>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> >>>
> >>> Hi guys,
> >>>
> >>> I have some quick questions regarding to Hadoop counter,
> >>>
> >>> Hadoop counter (customer defined) is global accessible (for both read
> and
> >>> write) for all Mappers and Reducers in a job?
> >>> What is the performance and best practices of using Hadoop counters? I
+
Lin Ma 2012-10-19, 16:09
-
Re: Hadoop counterHarsh J 2012-10-19, 16:19
Hi,
Inline.
On Fri, Oct 19, 2012 at 9:39 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
> Thanks for the great reply. Two basic questions,
>
> - Where the counters' value are stored for successful job? On JT?
Yes, they are ultimately stored at JT until the job is retired out of
heap memory (in which case, they get stored into the JobHistory
location and format).
> - Supposing a specific job A completed successfully and updated related
> counters, is it possible for another specific job B to read counters updated
> by previous job A? If yes, how?
Yes, possible, use the RunningJob object from the previous job (or
capture one) and query it. APIs you're interested in:
Grab a query-able object (RunningJob and/or a Job):
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobClient.html#getJob(org.apache.hadoop.mapred.JobID)
or http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Cluster.html#getJob(org.apache.hadoop.mapreduce.JobID)
Query counters:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
or http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
> regards,
> Lin
>
>
> On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Bejoy is almost right, except that counters are reported upon progress
>> of tasks itself (via TT heartbeats to JT actually), but the final
>> counter representation is computed only with successful task reports
>> the job received, not from any failed or killed ones.
>>
>> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:
>> > Hi Jay
>> >
>> > Counters are reported at the end of a task to JT. So if a task fails the
>> > counters from that task are not send to JT and hence won't be included
>> > in
>> > the final value of counters from that Job.
>> > Regards
>> > Bejoy KS
>> >
>> > Sent from handheld, please excuse typos.
>> > ________________________________
>> > From: Jay Vyas <[EMAIL PROTECTED]>
>> > Date: Fri, 19 Oct 2012 10:18:42 -0500
>> > To: <[EMAIL PROTECTED]>
>> > ReplyTo: [EMAIL PROTECTED]
>> > Subject: Re: Hadoop counter
>> >
>> > Ah this answers alot about why some of my dynamic counters never show up
>> > and
>> > i have to bite my nails waiting to see whats going on until the end of
>> > the
>> > job- thanks.
>> >
>> > Another question: what happens if a task fails ? What happen to the
>> > counters for it ? Do they dissappear into the ether? Or do they get
>> > merged
>> > in with the counters from other tasks?
>> >
>> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> And by default the number of counters is limited to 120 with the
>> >> mapreduce.job.counters.limit property.
>> >> They are useful for displaying short statistics about a job but should
>> >> not
>> >> be used for results (imho).
>> >> I know people may misuse them but I haven't tried so I wouldn't be able
>> >> to
>> >> list the caveats.
>> >>
>> >> Regards
>> >>
>> >> Bertrand
>> >>
>> >>
>> >> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel
>> >> <[EMAIL PROTECTED]>
>> >> wrote:
>> >>>
>> >>> As I understand it... each Task has its own counters and are
>> >>> independently updated. As they report back to the JT, they update the
>> >>> counter(s)' status.
>> >>> The JT then will aggregate them.
>> >>>
>> >>> In terms of performance, Counters take up some memory in the JT so
>> >>> while
>> >>> its OK to use them, if you abuse them, you can run in to issues.
>> >>> As to limits... I guess that will depend on the amount of memory on
>> >>> the
>> >>> JT machine, the size of the cluster (Number of TT) and the number of
>> >>> counters.
>> >>>
>> >>> In terms of global accessibility... Maybe.
>> >>>
>> >>> The reason I say maybe is that I'm not sure by what you mean by
>> >>> globally
>> >>> accessible.
>> >>> If a task creates and implements a dynamic counter... I know that it
>> >>> will
Harsh J
Inline.
On Fri, Oct 19, 2012 at 9:39 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
> Thanks for the great reply. Two basic questions,
>
> - Where the counters' value are stored for successful job? On JT?
Yes, they are ultimately stored at JT until the job is retired out of
heap memory (in which case, they get stored into the JobHistory
location and format).
> - Supposing a specific job A completed successfully and updated related
> counters, is it possible for another specific job B to read counters updated
> by previous job A? If yes, how?
Yes, possible, use the RunningJob object from the previous job (or
capture one) and query it. APIs you're interested in:
Grab a query-able object (RunningJob and/or a Job):
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobClient.html#getJob(org.apache.hadoop.mapred.JobID)
or http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Cluster.html#getJob(org.apache.hadoop.mapreduce.JobID)
Query counters:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
or http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
> regards,
> Lin
>
>
> On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Bejoy is almost right, except that counters are reported upon progress
>> of tasks itself (via TT heartbeats to JT actually), but the final
>> counter representation is computed only with successful task reports
>> the job received, not from any failed or killed ones.
>>
>> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:
>> > Hi Jay
>> >
>> > Counters are reported at the end of a task to JT. So if a task fails the
>> > counters from that task are not send to JT and hence won't be included
>> > in
>> > the final value of counters from that Job.
>> > Regards
>> > Bejoy KS
>> >
>> > Sent from handheld, please excuse typos.
>> > ________________________________
>> > From: Jay Vyas <[EMAIL PROTECTED]>
>> > Date: Fri, 19 Oct 2012 10:18:42 -0500
>> > To: <[EMAIL PROTECTED]>
>> > ReplyTo: [EMAIL PROTECTED]
>> > Subject: Re: Hadoop counter
>> >
>> > Ah this answers alot about why some of my dynamic counters never show up
>> > and
>> > i have to bite my nails waiting to see whats going on until the end of
>> > the
>> > job- thanks.
>> >
>> > Another question: what happens if a task fails ? What happen to the
>> > counters for it ? Do they dissappear into the ether? Or do they get
>> > merged
>> > in with the counters from other tasks?
>> >
>> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> And by default the number of counters is limited to 120 with the
>> >> mapreduce.job.counters.limit property.
>> >> They are useful for displaying short statistics about a job but should
>> >> not
>> >> be used for results (imho).
>> >> I know people may misuse them but I haven't tried so I wouldn't be able
>> >> to
>> >> list the caveats.
>> >>
>> >> Regards
>> >>
>> >> Bertrand
>> >>
>> >>
>> >> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel
>> >> <[EMAIL PROTECTED]>
>> >> wrote:
>> >>>
>> >>> As I understand it... each Task has its own counters and are
>> >>> independently updated. As they report back to the JT, they update the
>> >>> counter(s)' status.
>> >>> The JT then will aggregate them.
>> >>>
>> >>> In terms of performance, Counters take up some memory in the JT so
>> >>> while
>> >>> its OK to use them, if you abuse them, you can run in to issues.
>> >>> As to limits... I guess that will depend on the amount of memory on
>> >>> the
>> >>> JT machine, the size of the cluster (Number of TT) and the number of
>> >>> counters.
>> >>>
>> >>> In terms of global accessibility... Maybe.
>> >>>
>> >>> The reason I say maybe is that I'm not sure by what you mean by
>> >>> globally
>> >>> accessible.
>> >>> If a task creates and implements a dynamic counter... I know that it
>> >>> will
Harsh J
+
Harsh J 2012-10-19, 16:19
-
Re: Hadoop counterLin Ma 2012-10-19, 16:33
Hi Harsh,
Thanks for the brilliant reply.
For your comments -- "Yes, they are ultimately stored at JT until the job
is retired out of
heap memory (in which case, they get stored into the JobHistory
location and format).", does it mean only running job's counter will
consume JT memory, for completed job, counter will be stored in disk (I
think for "JobHistory location and format" is on disk?)?
regards,
Lin
On Sat, Oct 20, 2012 at 12:19 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Inline.
>
> On Fri, Oct 19, 2012 at 9:39 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
> > Hi Harsh,
> >
> > Thanks for the great reply. Two basic questions,
> >
> > - Where the counters' value are stored for successful job? On JT?
>
> Yes, they are ultimately stored at JT until the job is retired out of
> heap memory (in which case, they get stored into the JobHistory
> location and format).
>
> > - Supposing a specific job A completed successfully and updated related
> > counters, is it possible for another specific job B to read counters
> updated
> > by previous job A? If yes, how?
>
> Yes, possible, use the RunningJob object from the previous job (or
> capture one) and query it. APIs you're interested in:
>
> Grab a query-able object (RunningJob and/or a Job):
>
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobClient.html#getJob(org.apache.hadoop.mapred.JobID)
> or
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Cluster.html#getJob(org.apache.hadoop.mapreduce.JobID)
>
> Query counters:
>
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
> or
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
>
> > regards,
> > Lin
> >
> >
> > On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> Bejoy is almost right, except that counters are reported upon progress
> >> of tasks itself (via TT heartbeats to JT actually), but the final
> >> counter representation is computed only with successful task reports
> >> the job received, not from any failed or killed ones.
> >>
> >> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]>
> wrote:
> >> > Hi Jay
> >> >
> >> > Counters are reported at the end of a task to JT. So if a task fails
> the
> >> > counters from that task are not send to JT and hence won't be included
> >> > in
> >> > the final value of counters from that Job.
> >> > Regards
> >> > Bejoy KS
> >> >
> >> > Sent from handheld, please excuse typos.
> >> > ________________________________
> >> > From: Jay Vyas <[EMAIL PROTECTED]>
> >> > Date: Fri, 19 Oct 2012 10:18:42 -0500
> >> > To: <[EMAIL PROTECTED]>
> >> > ReplyTo: [EMAIL PROTECTED]
> >> > Subject: Re: Hadoop counter
> >> >
> >> > Ah this answers alot about why some of my dynamic counters never show
> up
> >> > and
> >> > i have to bite my nails waiting to see whats going on until the end of
> >> > the
> >> > job- thanks.
> >> >
> >> > Another question: what happens if a task fails ? What happen to the
> >> > counters for it ? Do they dissappear into the ether? Or do they get
> >> > merged
> >> > in with the counters from other tasks?
> >> >
> >> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]
> >
> >> > wrote:
> >> >>
> >> >> And by default the number of counters is limited to 120 with the
> >> >> mapreduce.job.counters.limit property.
> >> >> They are useful for displaying short statistics about a job but
> should
> >> >> not
> >> >> be used for results (imho).
> >> >> I know people may misuse them but I haven't tried so I wouldn't be
> able
> >> >> to
> >> >> list the caveats.
> >> >>
> >> >> Regards
> >> >>
> >> >> Bertrand
> >> >>
> >> >>
> >> >> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel
> >> >> <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >>>
> >> >>> As I understand it... each Task has its own counters and are
> >> >>> independently updated. As they report back to the JT, they update
Thanks for the brilliant reply.
For your comments -- "Yes, they are ultimately stored at JT until the job
is retired out of
heap memory (in which case, they get stored into the JobHistory
location and format).", does it mean only running job's counter will
consume JT memory, for completed job, counter will be stored in disk (I
think for "JobHistory location and format" is on disk?)?
regards,
Lin
On Sat, Oct 20, 2012 at 12:19 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Inline.
>
> On Fri, Oct 19, 2012 at 9:39 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
> > Hi Harsh,
> >
> > Thanks for the great reply. Two basic questions,
> >
> > - Where the counters' value are stored for successful job? On JT?
>
> Yes, they are ultimately stored at JT until the job is retired out of
> heap memory (in which case, they get stored into the JobHistory
> location and format).
>
> > - Supposing a specific job A completed successfully and updated related
> > counters, is it possible for another specific job B to read counters
> updated
> > by previous job A? If yes, how?
>
> Yes, possible, use the RunningJob object from the previous job (or
> capture one) and query it. APIs you're interested in:
>
> Grab a query-able object (RunningJob and/or a Job):
>
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobClient.html#getJob(org.apache.hadoop.mapred.JobID)
> or
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Cluster.html#getJob(org.apache.hadoop.mapreduce.JobID)
>
> Query counters:
>
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
> or
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
>
> > regards,
> > Lin
> >
> >
> > On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> Bejoy is almost right, except that counters are reported upon progress
> >> of tasks itself (via TT heartbeats to JT actually), but the final
> >> counter representation is computed only with successful task reports
> >> the job received, not from any failed or killed ones.
> >>
> >> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]>
> wrote:
> >> > Hi Jay
> >> >
> >> > Counters are reported at the end of a task to JT. So if a task fails
> the
> >> > counters from that task are not send to JT and hence won't be included
> >> > in
> >> > the final value of counters from that Job.
> >> > Regards
> >> > Bejoy KS
> >> >
> >> > Sent from handheld, please excuse typos.
> >> > ________________________________
> >> > From: Jay Vyas <[EMAIL PROTECTED]>
> >> > Date: Fri, 19 Oct 2012 10:18:42 -0500
> >> > To: <[EMAIL PROTECTED]>
> >> > ReplyTo: [EMAIL PROTECTED]
> >> > Subject: Re: Hadoop counter
> >> >
> >> > Ah this answers alot about why some of my dynamic counters never show
> up
> >> > and
> >> > i have to bite my nails waiting to see whats going on until the end of
> >> > the
> >> > job- thanks.
> >> >
> >> > Another question: what happens if a task fails ? What happen to the
> >> > counters for it ? Do they dissappear into the ether? Or do they get
> >> > merged
> >> > in with the counters from other tasks?
> >> >
> >> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]
> >
> >> > wrote:
> >> >>
> >> >> And by default the number of counters is limited to 120 with the
> >> >> mapreduce.job.counters.limit property.
> >> >> They are useful for displaying short statistics about a job but
> should
> >> >> not
> >> >> be used for results (imho).
> >> >> I know people may misuse them but I haven't tried so I wouldn't be
> able
> >> >> to
> >> >> list the caveats.
> >> >>
> >> >> Regards
> >> >>
> >> >> Bertrand
> >> >>
> >> >>
> >> >> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel
> >> >> <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >>>
> >> >>> As I understand it... each Task has its own counters and are
> >> >>> independently updated. As they report back to the JT, they update
+
Lin Ma 2012-10-19, 16:33
-
Re: Hadoop counterHarsh J 2012-10-19, 16:36
Hi,
There're essentially 4 lists on a live JT: Running, Completed,
Failed/Killed, Retired. The first 3 have counters in memory, the last
has it on disk. Completed and Failed/Killed jobs are sent to Retired
(on-disk persistence, and garbage collected out of memory), after a
default period of 24 hours post-finish time.
On Fri, Oct 19, 2012 at 10:03 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
> Thanks for the brilliant reply.
>
> For your comments -- "Yes, they are ultimately stored at JT until the job is
> retired out of
>
> heap memory (in which case, they get stored into the JobHistory
> location and format).", does it mean only running job's counter will consume
> JT memory, for completed job, counter will be stored in disk (I think for
> "JobHistory location and format" is on disk?)?
>
> regards,
> Lin
>
>
> On Sat, Oct 20, 2012 at 12:19 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> Inline.
>>
>> On Fri, Oct 19, 2012 at 9:39 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
>> > Hi Harsh,
>> >
>> > Thanks for the great reply. Two basic questions,
>> >
>> > - Where the counters' value are stored for successful job? On JT?
>>
>> Yes, they are ultimately stored at JT until the job is retired out of
>> heap memory (in which case, they get stored into the JobHistory
>> location and format).
>>
>> > - Supposing a specific job A completed successfully and updated related
>> > counters, is it possible for another specific job B to read counters
>> > updated
>> > by previous job A? If yes, how?
>>
>> Yes, possible, use the RunningJob object from the previous job (or
>> capture one) and query it. APIs you're interested in:
>>
>> Grab a query-able object (RunningJob and/or a Job):
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobClient.html#getJob(org.apache.hadoop.mapred.JobID)
>> or
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Cluster.html#getJob(org.apache.hadoop.mapreduce.JobID)
>>
>> Query counters:
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
>> or
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
>>
>> > regards,
>> > Lin
>> >
>> >
>> > On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Bejoy is almost right, except that counters are reported upon progress
>> >> of tasks itself (via TT heartbeats to JT actually), but the final
>> >> counter representation is computed only with successful task reports
>> >> the job received, not from any failed or killed ones.
>> >>
>> >> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hi Jay
>> >> >
>> >> > Counters are reported at the end of a task to JT. So if a task fails
>> >> > the
>> >> > counters from that task are not send to JT and hence won't be
>> >> > included
>> >> > in
>> >> > the final value of counters from that Job.
>> >> > Regards
>> >> > Bejoy KS
>> >> >
>> >> > Sent from handheld, please excuse typos.
>> >> > ________________________________
>> >> > From: Jay Vyas <[EMAIL PROTECTED]>
>> >> > Date: Fri, 19 Oct 2012 10:18:42 -0500
>> >> > To: <[EMAIL PROTECTED]>
>> >> > ReplyTo: [EMAIL PROTECTED]
>> >> > Subject: Re: Hadoop counter
>> >> >
>> >> > Ah this answers alot about why some of my dynamic counters never show
>> >> > up
>> >> > and
>> >> > i have to bite my nails waiting to see whats going on until the end
>> >> > of
>> >> > the
>> >> > job- thanks.
>> >> >
>> >> > Another question: what happens if a task fails ? What happen to the
>> >> > counters for it ? Do they dissappear into the ether? Or do they get
>> >> > merged
>> >> > in with the counters from other tasks?
>> >> >
>> >> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux
>> >> > <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> >>
>> >> >> And by default the number of counters is limited to 120 with the
>> >> >> mapreduce.job.counters.limit property.
>> >> >> They are useful for displaying short statistics about a job but
Harsh J
There're essentially 4 lists on a live JT: Running, Completed,
Failed/Killed, Retired. The first 3 have counters in memory, the last
has it on disk. Completed and Failed/Killed jobs are sent to Retired
(on-disk persistence, and garbage collected out of memory), after a
default period of 24 hours post-finish time.
On Fri, Oct 19, 2012 at 10:03 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
> Thanks for the brilliant reply.
>
> For your comments -- "Yes, they are ultimately stored at JT until the job is
> retired out of
>
> heap memory (in which case, they get stored into the JobHistory
> location and format).", does it mean only running job's counter will consume
> JT memory, for completed job, counter will be stored in disk (I think for
> "JobHistory location and format" is on disk?)?
>
> regards,
> Lin
>
>
> On Sat, Oct 20, 2012 at 12:19 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> Inline.
>>
>> On Fri, Oct 19, 2012 at 9:39 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
>> > Hi Harsh,
>> >
>> > Thanks for the great reply. Two basic questions,
>> >
>> > - Where the counters' value are stored for successful job? On JT?
>>
>> Yes, they are ultimately stored at JT until the job is retired out of
>> heap memory (in which case, they get stored into the JobHistory
>> location and format).
>>
>> > - Supposing a specific job A completed successfully and updated related
>> > counters, is it possible for another specific job B to read counters
>> > updated
>> > by previous job A? If yes, how?
>>
>> Yes, possible, use the RunningJob object from the previous job (or
>> capture one) and query it. APIs you're interested in:
>>
>> Grab a query-able object (RunningJob and/or a Job):
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobClient.html#getJob(org.apache.hadoop.mapred.JobID)
>> or
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Cluster.html#getJob(org.apache.hadoop.mapreduce.JobID)
>>
>> Query counters:
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
>> or
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()
>>
>> > regards,
>> > Lin
>> >
>> >
>> > On Fri, Oct 19, 2012 at 11:50 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Bejoy is almost right, except that counters are reported upon progress
>> >> of tasks itself (via TT heartbeats to JT actually), but the final
>> >> counter representation is computed only with successful task reports
>> >> the job received, not from any failed or killed ones.
>> >>
>> >> On Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hi Jay
>> >> >
>> >> > Counters are reported at the end of a task to JT. So if a task fails
>> >> > the
>> >> > counters from that task are not send to JT and hence won't be
>> >> > included
>> >> > in
>> >> > the final value of counters from that Job.
>> >> > Regards
>> >> > Bejoy KS
>> >> >
>> >> > Sent from handheld, please excuse typos.
>> >> > ________________________________
>> >> > From: Jay Vyas <[EMAIL PROTECTED]>
>> >> > Date: Fri, 19 Oct 2012 10:18:42 -0500
>> >> > To: <[EMAIL PROTECTED]>
>> >> > ReplyTo: [EMAIL PROTECTED]
>> >> > Subject: Re: Hadoop counter
>> >> >
>> >> > Ah this answers alot about why some of my dynamic counters never show
>> >> > up
>> >> > and
>> >> > i have to bite my nails waiting to see whats going on until the end
>> >> > of
>> >> > the
>> >> > job- thanks.
>> >> >
>> >> > Another question: what happens if a task fails ? What happen to the
>> >> > counters for it ? Do they dissappear into the ether? Or do they get
>> >> > merged
>> >> > in with the counters from other tasks?
>> >> >
>> >> > On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux
>> >> > <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> >>
>> >> >> And by default the number of counters is limited to 120 with the
>> >> >> mapreduce.job.counters.limit property.
>> >> >> They are useful for displaying short statistics about a job but
Harsh J
+
Harsh J 2012-10-19, 16:36
-
Re: Hadoop counterJay Vyas 2012-10-19, 16:02
That clarifies why counters can go down during a job. Very important
clarification because being able to rely on such "empemeral" counters is a
really important tool for realtime monitoring of failures: thanks guys
clarification because being able to rely on such "empemeral" counters is a
really important tool for realtime monitoring of failures: thanks guys
+
Jay Vyas 2012-10-19, 16:02
-
Hadoop counterLin Ma 2012-10-19, 10:50
Hi guys,
I have some quick questions regarding to Hadoop counter,
- Hadoop counter (customer defined) is global accessible (for both read
and write) for all Mappers and Reducers in a job?
- What is the performance and best practices of using Hadoop counters? I
am not sure if using Hadoop counters too heavy, there will be performance
downgrade to the whole job?
regards,
Lin
I have some quick questions regarding to Hadoop counter,
- Hadoop counter (customer defined) is global accessible (for both read
and write) for all Mappers and Reducers in a job?
- What is the performance and best practices of using Hadoop counters? I
am not sure if using Hadoop counters too heavy, there will be performance
downgrade to the whole job?
regards,
Lin
+
Lin Ma 2012-10-19, 10:50
-
Re: Hadoop counterMichael Segel 2012-10-19, 14:35
As I understand it... each Task has its own counters and are independently updated. As they report back to the JT, they update the counter(s)' status.
The JT then will aggregate them.
In terms of performance, Counters take up some memory in the JT so while its OK to use them, if you abuse them, you can run in to issues.
As to limits... I guess that will depend on the amount of memory on the JT machine, the size of the cluster (Number of TT) and the number of counters.
In terms of global accessibility... Maybe.
The reason I say maybe is that I'm not sure by what you mean by globally accessible.
If a task creates and implements a dynamic counter... I know that it will eventually be reflected in the JT. However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous. Not to mention that I don't believe that the counters are aggregated until the job ends. It would make sense that the JT maintains a unique counter for each task until the tasks complete. (If a task fails, it would have to delete the counters so that when the task is restarted the correct count is maintained. ) Note, I haven't looked at the source code so I am probably wrong.
HTH
Mike
On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
> Hadoop counter (customer defined) is global accessible (for both read and write) for all Mappers and Reducers in a job?
> What is the performance and best practices of using Hadoop counters? I am not sure if using Hadoop counters too heavy, there will be performance downgrade to the whole job?
> regards,
> Lin
The JT then will aggregate them.
In terms of performance, Counters take up some memory in the JT so while its OK to use them, if you abuse them, you can run in to issues.
As to limits... I guess that will depend on the amount of memory on the JT machine, the size of the cluster (Number of TT) and the number of counters.
In terms of global accessibility... Maybe.
The reason I say maybe is that I'm not sure by what you mean by globally accessible.
If a task creates and implements a dynamic counter... I know that it will eventually be reflected in the JT. However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous. Not to mention that I don't believe that the counters are aggregated until the job ends. It would make sense that the JT maintains a unique counter for each task until the tasks complete. (If a task fails, it would have to delete the counters so that when the task is restarted the correct count is maintained. ) Note, I haven't looked at the source code so I am probably wrong.
HTH
Mike
On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
> Hadoop counter (customer defined) is global accessible (for both read and write) for all Mappers and Reducers in a job?
> What is the performance and best practices of using Hadoop counters? I am not sure if using Hadoop counters too heavy, there will be performance downgrade to the whole job?
> regards,
> Lin
+
Michael Segel 2012-10-19, 14:35
-
Re: Hadoop counterLin Ma 2012-10-19, 16:27
Hi Mike,
Thanks for the detailed reply. Two quick questions/comments,
1. For "task", you mean a specific mapper instance, or a specific reducer
instance?
2. "However, I do not believe that a separate Task could connect with the
JT and see if the counter exists or if it could get a value or even an
accurate value since the updates are asynchronous." -- do you mean if a
mapper is updating custom counter ABC, and another mapper is updating the
same customer counter ABC, their counter values are updated independently
by different mappers, and will not published (aggregated) externally until
job completed successfully?
regards,
Lin
On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel
<[EMAIL PROTECTED]>wrote:
> As I understand it... each Task has its own counters and are independently
> updated. As they report back to the JT, they update the counter(s)' status.
> The JT then will aggregate them.
>
> In terms of performance, Counters take up some memory in the JT so while
> its OK to use them, if you abuse them, you can run in to issues.
> As to limits... I guess that will depend on the amount of memory on the JT
> machine, the size of the cluster (Number of TT) and the number of counters.
>
> In terms of global accessibility... Maybe.
>
> The reason I say maybe is that I'm not sure by what you mean by globally
> accessible.
> If a task creates and implements a dynamic counter... I know that it will
> eventually be reflected in the JT. However, I do not believe that a
> separate Task could connect with the JT and see if the counter exists or if
> it could get a value or even an accurate value since the updates are
> asynchronous. Not to mention that I don't believe that the counters are
> aggregated until the job ends. It would make sense that the JT maintains a
> unique counter for each task until the tasks complete. (If a task fails, it
> would have to delete the counters so that when the task is restarted the
> correct count is maintained. ) Note, I haven't looked at the source code
> so I am probably wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
>
> - Hadoop counter (customer defined) is global accessible (for both
> read and write) for all Mappers and Reducers in a job?
> - What is the performance and best practices of using Hadoop counters?
> I am not sure if using Hadoop counters too heavy, there will be performance
> downgrade to the whole job?
>
> regards,
> Lin
>
>
>
Thanks for the detailed reply. Two quick questions/comments,
1. For "task", you mean a specific mapper instance, or a specific reducer
instance?
2. "However, I do not believe that a separate Task could connect with the
JT and see if the counter exists or if it could get a value or even an
accurate value since the updates are asynchronous." -- do you mean if a
mapper is updating custom counter ABC, and another mapper is updating the
same customer counter ABC, their counter values are updated independently
by different mappers, and will not published (aggregated) externally until
job completed successfully?
regards,
Lin
On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel
<[EMAIL PROTECTED]>wrote:
> As I understand it... each Task has its own counters and are independently
> updated. As they report back to the JT, they update the counter(s)' status.
> The JT then will aggregate them.
>
> In terms of performance, Counters take up some memory in the JT so while
> its OK to use them, if you abuse them, you can run in to issues.
> As to limits... I guess that will depend on the amount of memory on the JT
> machine, the size of the cluster (Number of TT) and the number of counters.
>
> In terms of global accessibility... Maybe.
>
> The reason I say maybe is that I'm not sure by what you mean by globally
> accessible.
> If a task creates and implements a dynamic counter... I know that it will
> eventually be reflected in the JT. However, I do not believe that a
> separate Task could connect with the JT and see if the counter exists or if
> it could get a value or even an accurate value since the updates are
> asynchronous. Not to mention that I don't believe that the counters are
> aggregated until the job ends. It would make sense that the JT maintains a
> unique counter for each task until the tasks complete. (If a task fails, it
> would have to delete the counters so that when the task is restarted the
> correct count is maintained. ) Note, I haven't looked at the source code
> so I am probably wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
>
> - Hadoop counter (customer defined) is global accessible (for both
> read and write) for all Mappers and Reducers in a job?
> - What is the performance and best practices of using Hadoop counters?
> I am not sure if using Hadoop counters too heavy, there will be performance
> downgrade to the whole job?
>
> regards,
> Lin
>
>
>
+
Lin Ma 2012-10-19, 16:27
-
Re: Hadoop counterMichael Segel 2012-10-19, 16:42
On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi Mike,
>
> Thanks for the detailed reply. Two quick questions/comments,
>
> 1. For "task", you mean a specific mapper instance, or a specific reducer instance?
Either.
> 2. "However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous." -- do you mean if a mapper is updating custom counter ABC, and another mapper is updating the same customer counter ABC, their counter values are updated independently by different mappers, and will not published (aggregated) externally until job completed successfully?
>
I meant that if a Task created and updated a counter, a different Task has access to that counter.
To give you an example, if I want to count the number of quality errors and then fail after X number of errors, I can't use Global counters to do this.
> regards,
> Lin
>
> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
> As I understand it... each Task has its own counters and are independently updated. As they report back to the JT, they update the counter(s)' status.
> The JT then will aggregate them.
>
> In terms of performance, Counters take up some memory in the JT so while its OK to use them, if you abuse them, you can run in to issues.
> As to limits... I guess that will depend on the amount of memory on the JT machine, the size of the cluster (Number of TT) and the number of counters.
>
> In terms of global accessibility... Maybe.
>
> The reason I say maybe is that I'm not sure by what you mean by globally accessible.
> If a task creates and implements a dynamic counter... I know that it will eventually be reflected in the JT. However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous. Not to mention that I don't believe that the counters are aggregated until the job ends. It would make sense that the JT maintains a unique counter for each task until the tasks complete. (If a task fails, it would have to delete the counters so that when the task is restarted the correct count is maintained. ) Note, I haven't looked at the source code so I am probably wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
>> Hi guys,
>>
>> I have some quick questions regarding to Hadoop counter,
>>
>> Hadoop counter (customer defined) is global accessible (for both read and write) for all Mappers and Reducers in a job?
>> What is the performance and best practices of using Hadoop counters? I am not sure if using Hadoop counters too heavy, there will be performance downgrade to the whole job?
>> regards,
>> Lin
>
>
+
Michael Segel 2012-10-19, 16:42
-
Re: Hadoop counterLin Ma 2012-10-19, 16:56
Hi Mike,
Sorry I am a bit lost... As you are thinking faster than me. :-P
>From your this statement "It would make sense that the JT maintains a
unique counter for each task until the tasks complete." -- it seems each
task cannot see counters from each other, since JT maintains a unique
counter for each tasks;
>From your this comment "I meant that if a Task created and updated a
counter, a different Task has access to that counter. " -- it seems
different tasks could share/access the same counter.
Appreciate if you could help to clarify a bit.
regards,
Lin
On Sat, Oct 20, 2012 at 12:42 AM, Michael Segel
<[EMAIL PROTECTED]>wrote:
>
> On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi Mike,
>
> Thanks for the detailed reply. Two quick questions/comments,
>
> 1. For "task", you mean a specific mapper instance, or a specific reducer
> instance?
>
>
> Either.
>
> 2. "However, I do not believe that a separate Task could connect with the
> JT and see if the counter exists or if it could get a value or even an
> accurate value since the updates are asynchronous." -- do you mean if a
> mapper is updating custom counter ABC, and another mapper is updating the
> same customer counter ABC, their counter values are updated independently
> by different mappers, and will not published (aggregated) externally until
> job completed successfully?
>
> I meant that if a Task created and updated a counter, a different Task has
> access to that counter.
>
> To give you an example, if I want to count the number of quality errors
> and then fail after X number of errors, I can't use Global counters to do
> this.
>
> regards,
> Lin
>
> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <[EMAIL PROTECTED]
> > wrote:
>
>> As I understand it... each Task has its own counters and are
>> independently updated. As they report back to the JT, they update the
>> counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while
>> its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the
>> JT machine, the size of the cluster (Number of TT) and the number of
>> counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally
>> accessible.
>> If a task creates and implements a dynamic counter... I know that it will
>> eventually be reflected in the JT. However, I do not believe that a
>> separate Task could connect with the JT and see if the counter exists or if
>> it could get a value or even an accurate value since the updates are
>> asynchronous. Not to mention that I don't believe that the counters are
>> aggregated until the job ends. It would make sense that the JT maintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I haven't looked at the source code
>> so I am probably wrong.
>>
>> HTH
>> Mike
>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi guys,
>>
>> I have some quick questions regarding to Hadoop counter,
>>
>>
>> - Hadoop counter (customer defined) is global accessible (for both
>> read and write) for all Mappers and Reducers in a job?
>> - What is the performance and best practices of using Hadoop
>> counters? I am not sure if using Hadoop counters too heavy, there will be
>> performance downgrade to the whole job?
>>
>> regards,
>> Lin
>>
>>
>>
>
>
Sorry I am a bit lost... As you are thinking faster than me. :-P
>From your this statement "It would make sense that the JT maintains a
unique counter for each task until the tasks complete." -- it seems each
task cannot see counters from each other, since JT maintains a unique
counter for each tasks;
>From your this comment "I meant that if a Task created and updated a
counter, a different Task has access to that counter. " -- it seems
different tasks could share/access the same counter.
Appreciate if you could help to clarify a bit.
regards,
Lin
On Sat, Oct 20, 2012 at 12:42 AM, Michael Segel
<[EMAIL PROTECTED]>wrote:
>
> On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi Mike,
>
> Thanks for the detailed reply. Two quick questions/comments,
>
> 1. For "task", you mean a specific mapper instance, or a specific reducer
> instance?
>
>
> Either.
>
> 2. "However, I do not believe that a separate Task could connect with the
> JT and see if the counter exists or if it could get a value or even an
> accurate value since the updates are asynchronous." -- do you mean if a
> mapper is updating custom counter ABC, and another mapper is updating the
> same customer counter ABC, their counter values are updated independently
> by different mappers, and will not published (aggregated) externally until
> job completed successfully?
>
> I meant that if a Task created and updated a counter, a different Task has
> access to that counter.
>
> To give you an example, if I want to count the number of quality errors
> and then fail after X number of errors, I can't use Global counters to do
> this.
>
> regards,
> Lin
>
> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <[EMAIL PROTECTED]
> > wrote:
>
>> As I understand it... each Task has its own counters and are
>> independently updated. As they report back to the JT, they update the
>> counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while
>> its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the
>> JT machine, the size of the cluster (Number of TT) and the number of
>> counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally
>> accessible.
>> If a task creates and implements a dynamic counter... I know that it will
>> eventually be reflected in the JT. However, I do not believe that a
>> separate Task could connect with the JT and see if the counter exists or if
>> it could get a value or even an accurate value since the updates are
>> asynchronous. Not to mention that I don't believe that the counters are
>> aggregated until the job ends. It would make sense that the JT maintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I haven't looked at the source code
>> so I am probably wrong.
>>
>> HTH
>> Mike
>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi guys,
>>
>> I have some quick questions regarding to Hadoop counter,
>>
>>
>> - Hadoop counter (customer defined) is global accessible (for both
>> read and write) for all Mappers and Reducers in a job?
>> - What is the performance and best practices of using Hadoop
>> counters? I am not sure if using Hadoop counters too heavy, there will be
>> performance downgrade to the whole job?
>>
>> regards,
>> Lin
>>
>>
>>
>
>
+
Lin Ma 2012-10-19, 16:56
-
Re: Hadoop counterMichael Segel 2012-10-19, 21:05
Yeah, sorry...
I meant that if you were dynamically creating a counter foo in the Mapper task, then each mapper would be creating their own counter foo.
As the job runs, these counters will eventually be sent up to the JT. The job tracker would keep a separate counter for each task.
At the end, the final count is aggregated from the list of counters for foo.
I don't know how you can get a task to ask information from the Job Tracker on how things are going in other tasks. That is what I meant that you couldn't get information about the other counters or even the status of the other tasks running in the same job.
I didn't see anything in the APIs that allowed for that type of flow... Of course having said that... someone pops up with a way to do just that. ;-)
Does that clarify things?
-Mike
On Oct 19, 2012, at 11:56 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi Mike,
>
> Sorry I am a bit lost... As you are thinking faster than me. :-P
>
> From your this statement "It would make sense that the JT maintains a unique counter for each task until the tasks complete." -- it seems each task cannot see counters from each other, since JT maintains a unique counter for each tasks;
>
> From your this comment "I meant that if a Task created and updated a counter, a different Task has access to that counter. " -- it seems different tasks could share/access the same counter.
>
> Appreciate if you could help to clarify a bit.
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 12:42 AM, Michael Segel <[EMAIL PROTECTED]> wrote:
>
> On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
>> Hi Mike,
>>
>> Thanks for the detailed reply. Two quick questions/comments,
>>
>> 1. For "task", you mean a specific mapper instance, or a specific reducer instance?
>
> Either.
>
>> 2. "However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous." -- do you mean if a mapper is updating custom counter ABC, and another mapper is updating the same customer counter ABC, their counter values are updated independently by different mappers, and will not published (aggregated) externally until job completed successfully?
>>
> I meant that if a Task created and updated a counter, a different Task has access to that counter.
>
> To give you an example, if I want to count the number of quality errors and then fail after X number of errors, I can't use Global counters to do this.
>
>> regards,
>> Lin
>>
>> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>> As I understand it... each Task has its own counters and are independently updated. As they report back to the JT, they update the counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the JT machine, the size of the cluster (Number of TT) and the number of counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally accessible.
>> If a task creates and implements a dynamic counter... I know that it will eventually be reflected in the JT. However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous. Not to mention that I don't believe that the counters are aggregated until the job ends. It would make sense that the JT maintains a unique counter for each task until the tasks complete. (If a task fails, it would have to delete the counters so that when the task is restarted the correct count is maintained. ) Note, I haven't looked at the source code so I am probably wrong.
>>
>> HTH
>
I meant that if you were dynamically creating a counter foo in the Mapper task, then each mapper would be creating their own counter foo.
As the job runs, these counters will eventually be sent up to the JT. The job tracker would keep a separate counter for each task.
At the end, the final count is aggregated from the list of counters for foo.
I don't know how you can get a task to ask information from the Job Tracker on how things are going in other tasks. That is what I meant that you couldn't get information about the other counters or even the status of the other tasks running in the same job.
I didn't see anything in the APIs that allowed for that type of flow... Of course having said that... someone pops up with a way to do just that. ;-)
Does that clarify things?
-Mike
On Oct 19, 2012, at 11:56 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Hi Mike,
>
> Sorry I am a bit lost... As you are thinking faster than me. :-P
>
> From your this statement "It would make sense that the JT maintains a unique counter for each task until the tasks complete." -- it seems each task cannot see counters from each other, since JT maintains a unique counter for each tasks;
>
> From your this comment "I meant that if a Task created and updated a counter, a different Task has access to that counter. " -- it seems different tasks could share/access the same counter.
>
> Appreciate if you could help to clarify a bit.
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 12:42 AM, Michael Segel <[EMAIL PROTECTED]> wrote:
>
> On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
>> Hi Mike,
>>
>> Thanks for the detailed reply. Two quick questions/comments,
>>
>> 1. For "task", you mean a specific mapper instance, or a specific reducer instance?
>
> Either.
>
>> 2. "However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous." -- do you mean if a mapper is updating custom counter ABC, and another mapper is updating the same customer counter ABC, their counter values are updated independently by different mappers, and will not published (aggregated) externally until job completed successfully?
>>
> I meant that if a Task created and updated a counter, a different Task has access to that counter.
>
> To give you an example, if I want to count the number of quality errors and then fail after X number of errors, I can't use Global counters to do this.
>
>> regards,
>> Lin
>>
>> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>> As I understand it... each Task has its own counters and are independently updated. As they report back to the JT, they update the counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the JT machine, the size of the cluster (Number of TT) and the number of counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally accessible.
>> If a task creates and implements a dynamic counter... I know that it will eventually be reflected in the JT. However, I do not believe that a separate Task could connect with the JT and see if the counter exists or if it could get a value or even an accurate value since the updates are asynchronous. Not to mention that I don't believe that the counters are aggregated until the job ends. It would make sense that the JT maintains a unique counter for each task until the tasks complete. (If a task fails, it would have to delete the counters so that when the task is restarted the correct count is maintained. ) Note, I haven't looked at the source code so I am probably wrong.
>>
>> HTH
>
+
Michael Segel 2012-10-19, 21:05
-
Re: Hadoop counterLin Ma 2012-10-20, 03:27
Thanks for the detailed reply Mike, I learned a lot from the discussion.
- I just want to confirm with you that, supposing in the same job, when a
specific task completed (and counter is aggregated in JT after the task
completed from our discussion?), the other running task in the same job
cannot get the updated counter value from the previous completed task? I am
asking this because I am thinking whether I can use counter to share a
global value between tasks.
- If so, what is the traditional use case of counter, only use counter
values after the whole job completes?
BTW: appreciate if you could share me a few use cases from your experience
about how counters are used.
regards,
Lin
On Sat, Oct 20, 2012 at 5:05 AM, Michael Segel <[EMAIL PROTECTED]>wrote:
> Yeah, sorry...
>
> I meant that if you were dynamically creating a counter foo in the Mapper
> task, then each mapper would be creating their own counter foo.
> As the job runs, these counters will eventually be sent up to the JT. The
> job tracker would keep a separate counter for each task.
>
> At the end, the final count is aggregated from the list of counters for
> foo.
>
>
> I don't know how you can get a task to ask information from the Job
> Tracker on how things are going in other tasks. That is what I meant that
> you couldn't get information about the other counters or even the status of
> the other tasks running in the same job.
>
> I didn't see anything in the APIs that allowed for that type of flow... Of
> course having said that... someone pops up with a way to do just that. ;-)
>
>
> Does that clarify things?
>
> -Mike
>
>
> On Oct 19, 2012, at 11:56 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi Mike,
>
> Sorry I am a bit lost... As you are thinking faster than me. :-P
>
> From your this statement "It would make sense that the JT maintains a
> unique counter for each task until the tasks complete." -- it seems each
> task cannot see counters from each other, since JT maintains a unique
> counter for each tasks;
>
> From your this comment "I meant that if a Task created and updated a
> counter, a different Task has access to that counter. " -- it seems
> different tasks could share/access the same counter.
>
> Appreciate if you could help to clarify a bit.
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 12:42 AM, Michael Segel <[EMAIL PROTECTED]
> > wrote:
>
>>
>> On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi Mike,
>>
>> Thanks for the detailed reply. Two quick questions/comments,
>>
>> 1. For "task", you mean a specific mapper instance, or a specific reducer
>> instance?
>>
>>
>> Either.
>>
>> 2. "However, I do not believe that a separate Task could connect with the
>> JT and see if the counter exists or if it could get a value or even an
>> accurate value since the updates are asynchronous." -- do you mean if a
>> mapper is updating custom counter ABC, and another mapper is updating the
>> same customer counter ABC, their counter values are updated independently
>> by different mappers, and will not published (aggregated) externally until
>> job completed successfully?
>>
>> I meant that if a Task created and updated a counter, a different Task
>> has access to that counter.
>>
>> To give you an example, if I want to count the number of quality errors
>> and then fail after X number of errors, I can't use Global counters to do
>> this.
>>
>> regards,
>> Lin
>>
>> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <
>> [EMAIL PROTECTED]> wrote:
>>
>>> As I understand it... each Task has its own counters and are
>>> independently updated. As they report back to the JT, they update the
>>> counter(s)' status.
>>> The JT then will aggregate them.
>>>
>>> In terms of performance, Counters take up some memory in the JT so while
>>> its OK to use them, if you abuse them, you can run in to issues.
>>> As to limits... I guess that will depend on the amount of memory on the
>>> JT machine, the size of the cluster (Number of TT) and the number of
- I just want to confirm with you that, supposing in the same job, when a
specific task completed (and counter is aggregated in JT after the task
completed from our discussion?), the other running task in the same job
cannot get the updated counter value from the previous completed task? I am
asking this because I am thinking whether I can use counter to share a
global value between tasks.
- If so, what is the traditional use case of counter, only use counter
values after the whole job completes?
BTW: appreciate if you could share me a few use cases from your experience
about how counters are used.
regards,
Lin
On Sat, Oct 20, 2012 at 5:05 AM, Michael Segel <[EMAIL PROTECTED]>wrote:
> Yeah, sorry...
>
> I meant that if you were dynamically creating a counter foo in the Mapper
> task, then each mapper would be creating their own counter foo.
> As the job runs, these counters will eventually be sent up to the JT. The
> job tracker would keep a separate counter for each task.
>
> At the end, the final count is aggregated from the list of counters for
> foo.
>
>
> I don't know how you can get a task to ask information from the Job
> Tracker on how things are going in other tasks. That is what I meant that
> you couldn't get information about the other counters or even the status of
> the other tasks running in the same job.
>
> I didn't see anything in the APIs that allowed for that type of flow... Of
> course having said that... someone pops up with a way to do just that. ;-)
>
>
> Does that clarify things?
>
> -Mike
>
>
> On Oct 19, 2012, at 11:56 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi Mike,
>
> Sorry I am a bit lost... As you are thinking faster than me. :-P
>
> From your this statement "It would make sense that the JT maintains a
> unique counter for each task until the tasks complete." -- it seems each
> task cannot see counters from each other, since JT maintains a unique
> counter for each tasks;
>
> From your this comment "I meant that if a Task created and updated a
> counter, a different Task has access to that counter. " -- it seems
> different tasks could share/access the same counter.
>
> Appreciate if you could help to clarify a bit.
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 12:42 AM, Michael Segel <[EMAIL PROTECTED]
> > wrote:
>
>>
>> On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi Mike,
>>
>> Thanks for the detailed reply. Two quick questions/comments,
>>
>> 1. For "task", you mean a specific mapper instance, or a specific reducer
>> instance?
>>
>>
>> Either.
>>
>> 2. "However, I do not believe that a separate Task could connect with the
>> JT and see if the counter exists or if it could get a value or even an
>> accurate value since the updates are asynchronous." -- do you mean if a
>> mapper is updating custom counter ABC, and another mapper is updating the
>> same customer counter ABC, their counter values are updated independently
>> by different mappers, and will not published (aggregated) externally until
>> job completed successfully?
>>
>> I meant that if a Task created and updated a counter, a different Task
>> has access to that counter.
>>
>> To give you an example, if I want to count the number of quality errors
>> and then fail after X number of errors, I can't use Global counters to do
>> this.
>>
>> regards,
>> Lin
>>
>> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <
>> [EMAIL PROTECTED]> wrote:
>>
>>> As I understand it... each Task has its own counters and are
>>> independently updated. As they report back to the JT, they update the
>>> counter(s)' status.
>>> The JT then will aggregate them.
>>>
>>> In terms of performance, Counters take up some memory in the JT so while
>>> its OK to use them, if you abuse them, you can run in to issues.
>>> As to limits... I guess that will depend on the amount of memory on the
>>> JT machine, the size of the cluster (Number of TT) and the number of
+
Lin Ma 2012-10-20, 03:27
-
Re: Hadoop counterMichael Segel 2012-10-20, 07:12
On Oct 19, 2012, at 10:27 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Thanks for the detailed reply Mike, I learned a lot from the discussion.
>
> - I just want to confirm with you that, supposing in the same job, when a specific task completed (and counter is aggregated in JT after the task completed from our discussion?), the other running task in the same job cannot get the updated counter value from the previous completed task? I am asking this because I am thinking whether I can use counter to share a global value between tasks.
Yes that is correct.
While I haven't looked at YARN (M/R 2.0) , M/R 1.x doesn't have an easy way for a task to query the job tracker. This might have changed in YARN
> - If so, what is the traditional use case of counter, only use counter values after the whole job completes?
>
Yes the counters are used to provide data at the end of the job...
> BTW: appreciate if you could share me a few use cases from your experience about how counters are used.
>
Well you have your typical job data like the number of records processed, total number of bytes read, bytes written...
But suppose you wanted to do some quality control on your input.
So you need to keep a track on the count of bad records. If this job is part of a process, you may want to include business logic in your job to halt the job flow if X% of the records contain bad data.
Or your process takes input records and in processing them, they sort the records based on some characteristic and you want to count those sorted records as you processed them.
For a more concrete example, the Illinois Tollway has these 'fast pass' lanes where cars equipped with RFID tags can have the tolls automatically deducted from their accounts rather than pay the toll manually each time.
Suppose we wanted to determine how many cars in the 'Fast Pass' lanes are cheaters where they drive through the sensor and the sensor doesn't capture the RFID tag. (Note its possible that you have a false positive where the car has an RFID chip but doesn't trip the sensor.) Pushing the data in a map/reduce job would require the use of counters.
Does that help?
-Mike
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 5:05 AM, Michael Segel <[EMAIL PROTECTED]> wrote:
> Yeah, sorry...
>
> I meant that if you were dynamically creating a counter foo in the Mapper task, then each mapper would be creating their own counter foo.
> As the job runs, these counters will eventually be sent up to the JT. The job tracker would keep a separate counter for each task.
>
> At the end, the final count is aggregated from the list of counters for foo.
>
>
> I don't know how you can get a task to ask information from the Job Tracker on how things are going in other tasks. That is what I meant that you couldn't get information about the other counters or even the status of the other tasks running in the same job.
>
> I didn't see anything in the APIs that allowed for that type of flow... Of course having said that... someone pops up with a way to do just that. ;-)
>
>
> Does that clarify things?
>
> -Mike
>
>
> On Oct 19, 2012, at 11:56 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
>> Hi Mike,
>>
>> Sorry I am a bit lost... As you are thinking faster than me. :-P
>>
>> From your this statement "It would make sense that the JT maintains a unique counter for each task until the tasks complete." -- it seems each task cannot see counters from each other, since JT maintains a unique counter for each tasks;
>>
>> From your this comment "I meant that if a Task created and updated a counter, a different Task has access to that counter. " -- it seems different tasks could share/access the same counter.
>>
>> Appreciate if you could help to clarify a bit.
>>
>> regards,
>> Lin
>>
>> On Sat, Oct 20, 2012 at 12:42 AM, Michael Segel <[EMAIL PROTECTED]> wrote:
>>
>> On Oct 19, 2012, at 11:27 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Mike,
>>>
>>> Thanks for the detailed reply. Two quick questions/comments,
+
Michael Segel 2012-10-20, 07:12
-
Re: Hadoop counterLin Ma 2012-10-21, 06:45
Thanks for the detailed reply, Mike. Yes, my most confusion is resolved by
you. The last two questions (or comments) are used to confirm my
understanding is correct,
- is it normal use case or best practices for a job to consume/read the
counters from previous completed job in an automatic way? I ask this
because I am not sure whether the most use case of counter is human read
and manual analysis, other then using another job to automatic consume the
counters?
- I want to confirm my understanding is correct, when each task completes,
JT will aggregate/update the global counter values from the specific
counter values updated by the complete task, but never expose global
counters values until job completes? If it is correct, I am wondering why
JT doing aggregation each time when a task completes, other than doing a
one time aggregation when the job completes? Is there any design choice
reasons? thanks.
regards,
Lin
On Sat, Oct 20, 2012 at 3:12 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
> On Oct 19, 2012, at 10:27 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Thanks for the detailed reply Mike, I learned a lot from the discussion.
>
> - I just want to confirm with you that, supposing in the same job, when a
> specific task completed (and counter is aggregated in JT after the task
> completed from our discussion?), the other running task in the same job
> cannot get the updated counter value from the previous completed task? I am
> asking this because I am thinking whether I can use counter to share a
> global value between tasks.
>
>
> Yes that is correct.
> While I haven't looked at YARN (M/R 2.0) , M/R 1.x doesn't have an easy
> way for a task to query the job tracker. This might have changed in YARN
>
> - If so, what is the traditional use case of counter, only use counter
> values after the whole job completes?
>
> Yes the counters are used to provide data at the end of the job...
>
> BTW: appreciate if you could share me a few use cases from your experience
> about how counters are used.
>
> Well you have your typical job data like the number of records processed,
> total number of bytes read, bytes written...
>
> But suppose you wanted to do some quality control on your input.
> So you need to keep a track on the count of bad records. If this job is
> part of a process, you may want to include business logic in your job to
> halt the job flow if X% of the records contain bad data.
>
> Or your process takes input records and in processing them, they sort the
> records based on some characteristic and you want to count those sorted
> records as you processed them.
>
> For a more concrete example, the Illinois Tollway has these 'fast pass'
> lanes where cars equipped with RFID tags can have the tolls automatically
> deducted from their accounts rather than pay the toll manually each time.
>
> Suppose we wanted to determine how many cars in the 'Fast Pass' lanes are
> cheaters where they drive through the sensor and the sensor doesn't capture
> the RFID tag. (Note its possible that you have a false positive where the
> car has an RFID chip but doesn't trip the sensor.) Pushing the data in a
> map/reduce job would require the use of counters.
>
> Does that help?
>
> -Mike
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 5:05 AM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> Yeah, sorry...
>>
>> I meant that if you were dynamically creating a counter foo in the Mapper
>> task, then each mapper would be creating their own counter foo.
>> As the job runs, these counters will eventually be sent up to the JT. The
>> job tracker would keep a separate counter for each task.
>>
>> At the end, the final count is aggregated from the list of counters for
>> foo.
>>
>>
>> I don't know how you can get a task to ask information from the Job
>> Tracker on how things are going in other tasks. That is what I meant that
>> you couldn't get information about the other counters or even the status of
>> the other tasks running in the same job.
you. The last two questions (or comments) are used to confirm my
understanding is correct,
- is it normal use case or best practices for a job to consume/read the
counters from previous completed job in an automatic way? I ask this
because I am not sure whether the most use case of counter is human read
and manual analysis, other then using another job to automatic consume the
counters?
- I want to confirm my understanding is correct, when each task completes,
JT will aggregate/update the global counter values from the specific
counter values updated by the complete task, but never expose global
counters values until job completes? If it is correct, I am wondering why
JT doing aggregation each time when a task completes, other than doing a
one time aggregation when the job completes? Is there any design choice
reasons? thanks.
regards,
Lin
On Sat, Oct 20, 2012 at 3:12 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
> On Oct 19, 2012, at 10:27 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Thanks for the detailed reply Mike, I learned a lot from the discussion.
>
> - I just want to confirm with you that, supposing in the same job, when a
> specific task completed (and counter is aggregated in JT after the task
> completed from our discussion?), the other running task in the same job
> cannot get the updated counter value from the previous completed task? I am
> asking this because I am thinking whether I can use counter to share a
> global value between tasks.
>
>
> Yes that is correct.
> While I haven't looked at YARN (M/R 2.0) , M/R 1.x doesn't have an easy
> way for a task to query the job tracker. This might have changed in YARN
>
> - If so, what is the traditional use case of counter, only use counter
> values after the whole job completes?
>
> Yes the counters are used to provide data at the end of the job...
>
> BTW: appreciate if you could share me a few use cases from your experience
> about how counters are used.
>
> Well you have your typical job data like the number of records processed,
> total number of bytes read, bytes written...
>
> But suppose you wanted to do some quality control on your input.
> So you need to keep a track on the count of bad records. If this job is
> part of a process, you may want to include business logic in your job to
> halt the job flow if X% of the records contain bad data.
>
> Or your process takes input records and in processing them, they sort the
> records based on some characteristic and you want to count those sorted
> records as you processed them.
>
> For a more concrete example, the Illinois Tollway has these 'fast pass'
> lanes where cars equipped with RFID tags can have the tolls automatically
> deducted from their accounts rather than pay the toll manually each time.
>
> Suppose we wanted to determine how many cars in the 'Fast Pass' lanes are
> cheaters where they drive through the sensor and the sensor doesn't capture
> the RFID tag. (Note its possible that you have a false positive where the
> car has an RFID chip but doesn't trip the sensor.) Pushing the data in a
> map/reduce job would require the use of counters.
>
> Does that help?
>
> -Mike
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 5:05 AM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> Yeah, sorry...
>>
>> I meant that if you were dynamically creating a counter foo in the Mapper
>> task, then each mapper would be creating their own counter foo.
>> As the job runs, these counters will eventually be sent up to the JT. The
>> job tracker would keep a separate counter for each task.
>>
>> At the end, the final count is aggregated from the list of counters for
>> foo.
>>
>>
>> I don't know how you can get a task to ask information from the Job
>> Tracker on how things are going in other tasks. That is what I meant that
>> you couldn't get information about the other counters or even the status of
>> the other tasks running in the same job.
+
Lin Ma 2012-10-21, 06:45
-
Re: Hadoop counterMichael Segel 2012-10-21, 12:15
On Oct 21, 2012, at 1:45 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
> Thanks for the detailed reply, Mike. Yes, my most confusion is resolved by you. The last two questions (or comments) are used to confirm my understanding is correct,
>
> - is it normal use case or best practices for a job to consume/read the counters from previous completed job in an automatic way? I ask this because I am not sure whether the most use case of counter is human read and manual analysis, other then using another job to automatic consume the counters?
Lin,
Every job has a set of counters to maintain job statistics.
This is specifically for human analysis and to help understand what happened with your job.
It allows you to see how much data is read in by the job, how many records processed to be measured against how long the job took to complete. It also showed you how much data is written back out.
In addition to this, a set of use cases for counters in Hadoop center on quality control. Its normal to chain jobs together to form a job flow.
A typical use case for Hadoop is to pull data from various sources, combine them and do some process on them, resulting in a data set that gets sent to another system for visualization.
In this use case, there are usually data cleansing and validation jobs. As they run, its possible to track a number of defective records. At the end of that specific job, from the ToolRunner, or whichever job class you used to launch your job, you can then get these aggregated counters for the job and determine if the process passed or failed. Based on this, you can exit your program with either a success or failed flag. Job Flow control tools like Oozie can capture this and then decide to continue or to stop and alert an operator of an error.
> - I want to confirm my understanding is correct, when each task completes, JT will aggregate/update the global counter values from the specific counter values updated by the complete task, but never expose global counters values until job completes? If it is correct, I am wondering why JT doing aggregation each time when a task completes, other than doing a one time aggregation when the job completes? Is there any design choice reasons? thanks.
That's a good question. I haven't looked at the code, so I can't say definitively when the JT performs its aggregation. However, as the job runs and in process, we can look at the job tracker web page(s) and see the counter summary. This would imply that there has to be some aggregation occurring mid-flight. (It would be trivial to sum the list of counters periodically to update the job statistics.) Note too that if the JT web pages can show a counter, its possible to then write a monitoring tool that can monitor the job while running and then kill the job mid flight if a certain threshold of a counter is met.
That is to say you could in theory write a monitoring process and watch the counters. If lets say an error counter hits a predetermined threshold, you could then issue a 'hadoop job -kill <job-id>' command.
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 3:12 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>
> On Oct 19, 2012, at 10:27 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
>> Thanks for the detailed reply Mike, I learned a lot from the discussion.
>>
>> - I just want to confirm with you that, supposing in the same job, when a specific task completed (and counter is aggregated in JT after the task completed from our discussion?), the other running task in the same job cannot get the updated counter value from the previous completed task? I am asking this because I am thinking whether I can use counter to share a global value between tasks.
>
> Yes that is correct.
> While I haven't looked at YARN (M/R 2.0) , M/R 1.x doesn't have an easy way for a task to query the job tracker. This might have changed in YARN
>
>> - If so, what is the traditional use case of counter, only use counter values after the whole job completes?
+
Michael Segel 2012-10-21, 12:15
-
Re: Hadoop counterLin Ma 2012-10-22, 08:00
Thanks for the help so much, Mike. I learned a lot from this discussion.
So, the conclusion I learned from the discussion should be, since how/when
JT merge counter in the middle of the process of a job is undefined and
internal behavior, it is more reliable to read counter after the whole job
completes? Agree?
regards,
Lin
On Sun, Oct 21, 2012 at 8:15 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
> On Oct 21, 2012, at 1:45 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Thanks for the detailed reply, Mike. Yes, my most confusion is resolved by
> you. The last two questions (or comments) are used to confirm my
> understanding is correct,
>
> - is it normal use case or best practices for a job to consume/read the
> counters from previous completed job in an automatic way? I ask this
> because I am not sure whether the most use case of counter is human read
> and manual analysis, other then using another job to automatic consume the
> counters?
>
>
> Lin,
> Every job has a set of counters to maintain job statistics.
> This is specifically for human analysis and to help understand what
> happened with your job.
> It allows you to see how much data is read in by the job, how many records
> processed to be measured against how long the job took to complete. It
> also showed you how much data is written back out.
>
> In addition to this, a set of use cases for counters in Hadoop center on
> quality control. Its normal to chain jobs together to form a job flow.
> A typical use case for Hadoop is to pull data from various sources,
> combine them and do some process on them, resulting in a data set that gets
> sent to another system for visualization.
>
> In this use case, there are usually data cleansing and validation jobs. As
> they run, its possible to track a number of defective records. At the end
> of that specific job, from the ToolRunner, or whichever job class you used
> to launch your job, you can then get these aggregated counters for the job
> and determine if the process passed or failed. Based on this, you can exit
> your program with either a success or failed flag. Job Flow control tools
> like Oozie can capture this and then decide to continue or to stop and
> alert an operator of an error.
>
> - I want to confirm my understanding is correct, when each task completes,
> JT will aggregate/update the global counter values from the specific
> counter values updated by the complete task, but never expose global
> counters values until job completes? If it is correct, I am wondering why
> JT doing aggregation each time when a task completes, other than doing a
> one time aggregation when the job completes? Is there any design choice
> reasons? thanks.
>
>
> That's a good question. I haven't looked at the code, so I can't say
> definitively when the JT performs its aggregation. However, as the job runs
> and in process, we can look at the job tracker web page(s) and see the
> counter summary. This would imply that there has to be some aggregation
> occurring mid-flight. (It would be trivial to sum the list of counters
> periodically to update the job statistics.) Note too that if the JT web
> pages can show a counter, its possible to then write a monitoring tool that
> can monitor the job while running and then kill the job mid flight if a
> certain threshold of a counter is met.
>
> That is to say you could in theory write a monitoring process and watch
> the counters. If lets say an error counter hits a predetermined threshold,
> you could then issue a 'hadoop job -kill <job-id>' command.
>
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 3:12 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>>
>> On Oct 19, 2012, at 10:27 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Thanks for the detailed reply Mike, I learned a lot from the discussion.
>>
>> - I just want to confirm with you that, supposing in the same job, when a
>> specific task completed (and counter is aggregated in JT after the task
>> completed from our discussion?), the other running task in the same job
So, the conclusion I learned from the discussion should be, since how/when
JT merge counter in the middle of the process of a job is undefined and
internal behavior, it is more reliable to read counter after the whole job
completes? Agree?
regards,
Lin
On Sun, Oct 21, 2012 at 8:15 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
> On Oct 21, 2012, at 1:45 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Thanks for the detailed reply, Mike. Yes, my most confusion is resolved by
> you. The last two questions (or comments) are used to confirm my
> understanding is correct,
>
> - is it normal use case or best practices for a job to consume/read the
> counters from previous completed job in an automatic way? I ask this
> because I am not sure whether the most use case of counter is human read
> and manual analysis, other then using another job to automatic consume the
> counters?
>
>
> Lin,
> Every job has a set of counters to maintain job statistics.
> This is specifically for human analysis and to help understand what
> happened with your job.
> It allows you to see how much data is read in by the job, how many records
> processed to be measured against how long the job took to complete. It
> also showed you how much data is written back out.
>
> In addition to this, a set of use cases for counters in Hadoop center on
> quality control. Its normal to chain jobs together to form a job flow.
> A typical use case for Hadoop is to pull data from various sources,
> combine them and do some process on them, resulting in a data set that gets
> sent to another system for visualization.
>
> In this use case, there are usually data cleansing and validation jobs. As
> they run, its possible to track a number of defective records. At the end
> of that specific job, from the ToolRunner, or whichever job class you used
> to launch your job, you can then get these aggregated counters for the job
> and determine if the process passed or failed. Based on this, you can exit
> your program with either a success or failed flag. Job Flow control tools
> like Oozie can capture this and then decide to continue or to stop and
> alert an operator of an error.
>
> - I want to confirm my understanding is correct, when each task completes,
> JT will aggregate/update the global counter values from the specific
> counter values updated by the complete task, but never expose global
> counters values until job completes? If it is correct, I am wondering why
> JT doing aggregation each time when a task completes, other than doing a
> one time aggregation when the job completes? Is there any design choice
> reasons? thanks.
>
>
> That's a good question. I haven't looked at the code, so I can't say
> definitively when the JT performs its aggregation. However, as the job runs
> and in process, we can look at the job tracker web page(s) and see the
> counter summary. This would imply that there has to be some aggregation
> occurring mid-flight. (It would be trivial to sum the list of counters
> periodically to update the job statistics.) Note too that if the JT web
> pages can show a counter, its possible to then write a monitoring tool that
> can monitor the job while running and then kill the job mid flight if a
> certain threshold of a counter is met.
>
> That is to say you could in theory write a monitoring process and watch
> the counters. If lets say an error counter hits a predetermined threshold,
> you could then issue a 'hadoop job -kill <job-id>' command.
>
>
> regards,
> Lin
>
> On Sat, Oct 20, 2012 at 3:12 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>>
>> On Oct 19, 2012, at 10:27 PM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Thanks for the detailed reply Mike, I learned a lot from the discussion.
>>
>> - I just want to confirm with you that, supposing in the same job, when a
>> specific task completed (and counter is aggregated in JT after the task
>> completed from our discussion?), the other running task in the same job
+
Lin Ma 2012-10-22, 08:00
-
Re: Hadoop counterBertrand Dechoux 2012-10-19, 14:50
And by default the number of counters is limited to 120 with the
mapreduce.job.counters.limit property.
They are useful for displaying short statistics about a job but should not
be used for results (imho).
I know people may misuse them but I haven't tried so I wouldn't be able to
list the caveats.
Regards
Bertrand
On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
> As I understand it... each Task has its own counters and are independently
> updated. As they report back to the JT, they update the counter(s)' status.
> The JT then will aggregate them.
>
> In terms of performance, Counters take up some memory in the JT so while
> its OK to use them, if you abuse them, you can run in to issues.
> As to limits... I guess that will depend on the amount of memory on the JT
> machine, the size of the cluster (Number of TT) and the number of counters.
>
> In terms of global accessibility... Maybe.
>
> The reason I say maybe is that I'm not sure by what you mean by globally
> accessible.
> If a task creates and implements a dynamic counter... I know that it will
> eventually be reflected in the JT. However, I do not believe that a
> separate Task could connect with the JT and see if the counter exists or if
> it could get a value or even an accurate value since the updates are
> asynchronous. Not to mention that I don't believe that the counters are
> aggregated until the job ends. It would make sense that the JT maintains a
> unique counter for each task until the tasks complete. (If a task fails, it
> would have to delete the counters so that when the task is restarted the
> correct count is maintained. ) Note, I haven't looked at the source code
> so I am probably wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
>
> - Hadoop counter (customer defined) is global accessible (for both
> read and write) for all Mappers and Reducers in a job?
> - What is the performance and best practices of using Hadoop counters?
> I am not sure if using Hadoop counters too heavy, there will be performance
> downgrade to the whole job?
>
> regards,
> Lin
>
>
>
--
Bertrand Dechoux
mapreduce.job.counters.limit property.
They are useful for displaying short statistics about a job but should not
be used for results (imho).
I know people may misuse them but I haven't tried so I wouldn't be able to
list the caveats.
Regards
Bertrand
On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
> As I understand it... each Task has its own counters and are independently
> updated. As they report back to the JT, they update the counter(s)' status.
> The JT then will aggregate them.
>
> In terms of performance, Counters take up some memory in the JT so while
> its OK to use them, if you abuse them, you can run in to issues.
> As to limits... I guess that will depend on the amount of memory on the JT
> machine, the size of the cluster (Number of TT) and the number of counters.
>
> In terms of global accessibility... Maybe.
>
> The reason I say maybe is that I'm not sure by what you mean by globally
> accessible.
> If a task creates and implements a dynamic counter... I know that it will
> eventually be reflected in the JT. However, I do not believe that a
> separate Task could connect with the JT and see if the counter exists or if
> it could get a value or even an accurate value since the updates are
> asynchronous. Not to mention that I don't believe that the counters are
> aggregated until the job ends. It would make sense that the JT maintains a
> unique counter for each task until the tasks complete. (If a task fails, it
> would have to delete the counters so that when the task is restarted the
> correct count is maintained. ) Note, I haven't looked at the source code
> so I am probably wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
>
> - Hadoop counter (customer defined) is global accessible (for both
> read and write) for all Mappers and Reducers in a job?
> - What is the performance and best practices of using Hadoop counters?
> I am not sure if using Hadoop counters too heavy, there will be performance
> downgrade to the whole job?
>
> regards,
> Lin
>
>
>
--
Bertrand Dechoux
+
Bertrand Dechoux 2012-10-19, 14:50
-
Re: Hadoop counterJay Vyas 2012-10-19, 15:18
Ah this answers alot about why some of my dynamic counters never show up
and i have to bite my nails waiting to see whats going on until the end of
the job- thanks.
Another question: what happens if a task fails ? What happen to the
counters for it ? Do they dissappear into the ether? Or do they get merged
in with the counters from other tasks?
On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
> And by default the number of counters is limited to 120 with the
> mapreduce.job.counters.limit property.
> They are useful for displaying short statistics about a job but should not
> be used for results (imho).
> I know people may misuse them but I haven't tried so I wouldn't be able to
> list the caveats.
>
> Regards
>
> Bertrand
>
>
> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> As I understand it... each Task has its own counters and are
>> independently updated. As they report back to the JT, they update the
>> counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while
>> its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the
>> JT machine, the size of the cluster (Number of TT) and the number of
>> counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally
>> accessible.
>> If a task creates and implements a dynamic counter... I know that it will
>> eventually be reflected in the JT. However, I do not believe that a
>> separate Task could connect with the JT and see if the counter exists or if
>> it could get a value or even an accurate value since the updates are
>> asynchronous. Not to mention that I don't believe that the counters are
>> aggregated until the job ends. It would make sense that the JT maintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I haven't looked at the source code
>> so I am probably wrong.
>>
>> HTH
>> Mike
>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi guys,
>>
>> I have some quick questions regarding to Hadoop counter,
>>
>>
>> - Hadoop counter (customer defined) is global accessible (for both
>> read and write) for all Mappers and Reducers in a job?
>> - What is the performance and best practices of using Hadoop
>> counters? I am not sure if using Hadoop counters too heavy, there will be
>> performance downgrade to the whole job?
>>
>> regards,
>> Lin
>>
>>
>>
>
>
> --
> Bertrand Dechoux
>
--
Jay Vyas
http://jayunit100.blogspot.com
and i have to bite my nails waiting to see whats going on until the end of
the job- thanks.
Another question: what happens if a task fails ? What happen to the
counters for it ? Do they dissappear into the ether? Or do they get merged
in with the counters from other tasks?
On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
> And by default the number of counters is limited to 120 with the
> mapreduce.job.counters.limit property.
> They are useful for displaying short statistics about a job but should not
> be used for results (imho).
> I know people may misuse them but I haven't tried so I wouldn't be able to
> list the caveats.
>
> Regards
>
> Bertrand
>
>
> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> As I understand it... each Task has its own counters and are
>> independently updated. As they report back to the JT, they update the
>> counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while
>> its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the
>> JT machine, the size of the cluster (Number of TT) and the number of
>> counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally
>> accessible.
>> If a task creates and implements a dynamic counter... I know that it will
>> eventually be reflected in the JT. However, I do not believe that a
>> separate Task could connect with the JT and see if the counter exists or if
>> it could get a value or even an accurate value since the updates are
>> asynchronous. Not to mention that I don't believe that the counters are
>> aggregated until the job ends. It would make sense that the JT maintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I haven't looked at the source code
>> so I am probably wrong.
>>
>> HTH
>> Mike
>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi guys,
>>
>> I have some quick questions regarding to Hadoop counter,
>>
>>
>> - Hadoop counter (customer defined) is global accessible (for both
>> read and write) for all Mappers and Reducers in a job?
>> - What is the performance and best practices of using Hadoop
>> counters? I am not sure if using Hadoop counters too heavy, there will be
>> performance downgrade to the whole job?
>>
>> regards,
>> Lin
>>
>>
>>
>
>
> --
> Bertrand Dechoux
>
--
Jay Vyas
http://jayunit100.blogspot.com
+
Jay Vyas 2012-10-19, 15:18
-
Re: Hadoop counterBejoy KS 2012-10-19, 15:21
Hi Jay
Counters are reported at the end of a task to JT. So if a task fails the counters from that task are not send to JT and hence won't be included in the final value of counters from that Job.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Jay Vyas <[EMAIL PROTECTED]>
Date: Fri, 19 Oct 2012 10:18:42
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Hadoop counter
Ah this answers alot about why some of my dynamic counters never show up
and i have to bite my nails waiting to see whats going on until the end of
the job- thanks.
Another question: what happens if a task fails ? What happen to the
counters for it ? Do they dissappear into the ether? Or do they get merged
in with the counters from other tasks?
On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
> And by default the number of counters is limited to 120 with the
> mapreduce.job.counters.limit property.
> They are useful for displaying short statistics about a job but should not
> be used for results (imho).
> I know people may misuse them but I haven't tried so I wouldn't be able to
> list the caveats.
>
> Regards
>
> Bertrand
>
>
> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> As I understand it... each Task has its own counters and are
>> independently updated. As they report back to the JT, they update the
>> counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while
>> its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the
>> JT machine, the size of the cluster (Number of TT) and the number of
>> counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally
>> accessible.
>> If a task creates and implements a dynamic counter... I know that it will
>> eventually be reflected in the JT. However, I do not believe that a
>> separate Task could connect with the JT and see if the counter exists or if
>> it could get a value or even an accurate value since the updates are
>> asynchronous. Not to mention that I don't believe that the counters are
>> aggregated until the job ends. It would make sense that the JT maintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I haven't looked at the source code
>> so I am probably wrong.
>>
>> HTH
>> Mike
>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi guys,
>>
>> I have some quick questions regarding to Hadoop counter,
>>
>>
>> - Hadoop counter (customer defined) is global accessible (for both
>> read and write) for all Mappers and Reducers in a job?
>> - What is the performance and best practices of using Hadoop
>> counters? I am not sure if using Hadoop counters too heavy, there will be
>> performance downgrade to the whole job?
>>
>> regards,
>> Lin
>>
>>
>>
>
>
> --
> Bertrand Dechoux
>
--
Jay Vyas
http://jayunit100.blogspot.com
Counters are reported at the end of a task to JT. So if a task fails the counters from that task are not send to JT and hence won't be included in the final value of counters from that Job.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Jay Vyas <[EMAIL PROTECTED]>
Date: Fri, 19 Oct 2012 10:18:42
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Hadoop counter
Ah this answers alot about why some of my dynamic counters never show up
and i have to bite my nails waiting to see whats going on until the end of
the job- thanks.
Another question: what happens if a task fails ? What happen to the
counters for it ? Do they dissappear into the ether? Or do they get merged
in with the counters from other tasks?
On Fri, Oct 19, 2012 at 9:50 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
> And by default the number of counters is limited to 120 with the
> mapreduce.job.counters.limit property.
> They are useful for displaying short statistics about a job but should not
> be used for results (imho).
> I know people may misuse them but I haven't tried so I wouldn't be able to
> list the caveats.
>
> Regards
>
> Bertrand
>
>
> On Fri, Oct 19, 2012 at 4:35 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> As I understand it... each Task has its own counters and are
>> independently updated. As they report back to the JT, they update the
>> counter(s)' status.
>> The JT then will aggregate them.
>>
>> In terms of performance, Counters take up some memory in the JT so while
>> its OK to use them, if you abuse them, you can run in to issues.
>> As to limits... I guess that will depend on the amount of memory on the
>> JT machine, the size of the cluster (Number of TT) and the number of
>> counters.
>>
>> In terms of global accessibility... Maybe.
>>
>> The reason I say maybe is that I'm not sure by what you mean by globally
>> accessible.
>> If a task creates and implements a dynamic counter... I know that it will
>> eventually be reflected in the JT. However, I do not believe that a
>> separate Task could connect with the JT and see if the counter exists or if
>> it could get a value or even an accurate value since the updates are
>> asynchronous. Not to mention that I don't believe that the counters are
>> aggregated until the job ends. It would make sense that the JT maintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I haven't looked at the source code
>> so I am probably wrong.
>>
>> HTH
>> Mike
>> On Oct 19, 2012, at 5:50 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Hi guys,
>>
>> I have some quick questions regarding to Hadoop counter,
>>
>>
>> - Hadoop counter (customer defined) is global accessible (for both
>> read and write) for all Mappers and Reducers in a job?
>> - What is the performance and best practices of using Hadoop
>> counters? I am not sure if using Hadoop counters too heavy, there will be
>> performance downgrade to the whole job?
>>
>> regards,
>> Lin
>>
>>
>>
>
>
> --
> Bertrand Dechoux
>
--
Jay Vyas
http://jayunit100.blogspot.com
+
Bejoy KS 2012-10-19, 15:21
-
Re: Hadoop counterLin Ma 2012-10-23, 07:12
Thanks for the long discussion Mile. Learned a lot from you.
regards,
Lin
On Tue, Oct 23, 2012 at 11:57 AM, Michael Segel
<[EMAIL PROTECTED]>wrote:
> Yup.
> The counters at the end of the job are the most accurate.
>
> On Oct 22, 2012, at 3:00 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Thanks for the help so much, Mike. I learned a lot from this discussion.
>
> So, the conclusion I learned from the discussion should be, since how/when
> JT merge counter in the middle of the process of a job is undefined and
> internal behavior, it is more reliable to read counter after the whole job
> completes? Agree?
>
> regards,
> Lin
>
> On Sun, Oct 21, 2012 at 8:15 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>>
>> On Oct 21, 2012, at 1:45 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Thanks for the detailed reply, Mike. Yes, my most confusion is resolved
>> by you. The last two questions (or comments) are used to confirm my
>> understanding is correct,
>>
>> - is it normal use case or best practices for a job to consume/read the
>> counters from previous completed job in an automatic way? I ask this
>> because I am not sure whether the most use case of counter is human read
>> and manual analysis, other then using another job to automatic consume the
>> counters?
>>
>>
>> Lin,
>> Every job has a set of counters to maintain job statistics.
>> This is specifically for human analysis and to help understand what
>> happened with your job.
>> It allows you to see how much data is read in by the job, how many
>> records processed to be measured against how long the job took to complete.
>> It also showed you how much data is written back out.
>>
>> In addition to this, a set of use cases for counters in Hadoop center on
>> quality control. Its normal to chain jobs together to form a job flow.
>> A typical use case for Hadoop is to pull data from various sources,
>> combine them and do some process on them, resulting in a data set that gets
>> sent to another system for visualization.
>>
>> In this use case, there are usually data cleansing and validation jobs.
>> As they run, its possible to track a number of defective records. At the
>> end of that specific job, from the ToolRunner, or whichever job class you
>> used to launch your job, you can then get these aggregated counters for the
>> job and determine if the process passed or failed. Based on this, you can
>> exit your program with either a success or failed flag. Job Flow control
>> tools like Oozie can capture this and then decide to continue or to stop
>> and alert an operator of an error.
>>
>> - I want to confirm my understanding is correct, when each task
>> completes, JT will aggregate/update the global counter values from the
>> specific counter values updated by the complete task, but never expose
>> global counters values until job completes? If it is correct, I am
>> wondering why JT doing aggregation each time when a task completes, other
>> than doing a one time aggregation when the job completes? Is there any
>> design choice reasons? thanks.
>>
>>
>> That's a good question. I haven't looked at the code, so I can't say
>> definitively when the JT performs its aggregation. However, as the job runs
>> and in process, we can look at the job tracker web page(s) and see the
>> counter summary. This would imply that there has to be some aggregation
>> occurring mid-flight. (It would be trivial to sum the list of counters
>> periodically to update the job statistics.) Note too that if the JT web
>> pages can show a counter, its possible to then write a monitoring tool that
>> can monitor the job while running and then kill the job mid flight if a
>> certain threshold of a counter is met.
>>
>> That is to say you could in theory write a monitoring process and watch
>> the counters. If lets say an error counter hits a predetermined threshold,
>> you could then issue a 'hadoop job -kill <job-id>' command.
>>
>>
>> regards,
>> Lin
>>
>> On Sat, Oct 20, 2012 at 3:12 PM, Michael Segel <[EMAIL PROTECTED]
regards,
Lin
On Tue, Oct 23, 2012 at 11:57 AM, Michael Segel
<[EMAIL PROTECTED]>wrote:
> Yup.
> The counters at the end of the job are the most accurate.
>
> On Oct 22, 2012, at 3:00 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>
> Thanks for the help so much, Mike. I learned a lot from this discussion.
>
> So, the conclusion I learned from the discussion should be, since how/when
> JT merge counter in the middle of the process of a job is undefined and
> internal behavior, it is more reliable to read counter after the whole job
> completes? Agree?
>
> regards,
> Lin
>
> On Sun, Oct 21, 2012 at 8:15 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>>
>> On Oct 21, 2012, at 1:45 AM, Lin Ma <[EMAIL PROTECTED]> wrote:
>>
>> Thanks for the detailed reply, Mike. Yes, my most confusion is resolved
>> by you. The last two questions (or comments) are used to confirm my
>> understanding is correct,
>>
>> - is it normal use case or best practices for a job to consume/read the
>> counters from previous completed job in an automatic way? I ask this
>> because I am not sure whether the most use case of counter is human read
>> and manual analysis, other then using another job to automatic consume the
>> counters?
>>
>>
>> Lin,
>> Every job has a set of counters to maintain job statistics.
>> This is specifically for human analysis and to help understand what
>> happened with your job.
>> It allows you to see how much data is read in by the job, how many
>> records processed to be measured against how long the job took to complete.
>> It also showed you how much data is written back out.
>>
>> In addition to this, a set of use cases for counters in Hadoop center on
>> quality control. Its normal to chain jobs together to form a job flow.
>> A typical use case for Hadoop is to pull data from various sources,
>> combine them and do some process on them, resulting in a data set that gets
>> sent to another system for visualization.
>>
>> In this use case, there are usually data cleansing and validation jobs.
>> As they run, its possible to track a number of defective records. At the
>> end of that specific job, from the ToolRunner, or whichever job class you
>> used to launch your job, you can then get these aggregated counters for the
>> job and determine if the process passed or failed. Based on this, you can
>> exit your program with either a success or failed flag. Job Flow control
>> tools like Oozie can capture this and then decide to continue or to stop
>> and alert an operator of an error.
>>
>> - I want to confirm my understanding is correct, when each task
>> completes, JT will aggregate/update the global counter values from the
>> specific counter values updated by the complete task, but never expose
>> global counters values until job completes? If it is correct, I am
>> wondering why JT doing aggregation each time when a task completes, other
>> than doing a one time aggregation when the job completes? Is there any
>> design choice reasons? thanks.
>>
>>
>> That's a good question. I haven't looked at the code, so I can't say
>> definitively when the JT performs its aggregation. However, as the job runs
>> and in process, we can look at the job tracker web page(s) and see the
>> counter summary. This would imply that there has to be some aggregation
>> occurring mid-flight. (It would be trivial to sum the list of counters
>> periodically to update the job statistics.) Note too that if the JT web
>> pages can show a counter, its possible to then write a monitoring tool that
>> can monitor the job while running and then kill the job mid flight if a
>> certain threshold of a counter is met.
>>
>> That is to say you could in theory write a monitoring process and watch
>> the counters. If lets say an error counter hits a predetermined threshold,
>> you could then issue a 'hadoop job -kill <job-id>' command.
>>
>>
>> regards,
>> Lin
>>
>> On Sat, Oct 20, 2012 at 3:12 PM, Michael Segel <[EMAIL PROTECTED]
+
Lin Ma 2012-10-23, 07:12
