-Re: Custom Hadoop metrics in Map Reduce jobs
Dino Kečo 2011-11-02, 10:29
Yes I am looking for much detailed information beyond counters. Counters
are pretty cool but I would like to gather more information related to
record (like processing time in some part of code, some business logic
attributes and etc.) So I was hoping to use hadoop metrics to save that
into file or table and use it for reporting.
Thanks for info, please share your blog links so we can see something new :)
On Wed, Nov 2, 2011 at 10:59 AM, Daniel Iancu <[EMAIL PROTECTED]> wrote:
> Hi Dino
> It can't be used effectively to monitor MR mappers since tasks on a node
> run as separate JVMs and each task will overwrite the same metric published
> by other tasks. You cannot aggregate values from running tasks before
> publish them because tasks don't communicate with each other.
> If you publish a metric per task, this will give you tones of graphs in
> On other hand I extended it the API and use it to monitor our client
> (webapp) and it works fine and provide valuable information. I have started
> a blog post to document this, will be ready soon, I'll add the link here if
> you need some examples. I wonder why this API is not a separate project, it
> can be reused in apps that don't need Hadoop. I could not find any java
> client API to publish metrics in Ganglia.
> On 11/02/2011 09:44 AM, Dino Kečo wrote:
> Hi Hadoop users,
> I have been reading about Hadoop metrics framework, and I was wondering
> is it possible to create custom metrics for specific job.
> In my use case I want to capture some specific information per record
> which is processed. So I was thinking to create custom code in my Hadoop
> job to send these information using Hadoop metrics. Is this possible to
> implement ? If so can anyone share any example how things should be done?