Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Tools for extracting data from hadoop logs


+
bharath vissapragada 2012-10-30, 01:48
+
Binglin Chang 2012-10-30, 03:24
+
bharath vissapragada 2012-10-30, 05:03
+
Raj Vishwanathan 2012-10-30, 05:21
Copy link to this message
-
Re: Tools for extracting data from hadoop logs
Much useful one thanks Binglin for sharing it!

Cheers!
Manoj.

On Tue, Oct 30, 2012 at 8:54 AM, Binglin Chang <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I think you want to analyze hadoop job logs in jobtracker history folder?
> These logs are in a centralized folder and don't need tools like flume or
> scribe to gather them.
> I used to write a simple python script to parse those log files, and
> generate csv/json reports, basically you can use it to get execution time,
> counter, status of job, taks, attempts, maybe you can modify it to meet you
> needs.
>
> Thanks,
> Binglin
>
>
> On Tue, Oct 30, 2012 at 9:48 AM, bharath vissapragada <
> [EMAIL PROTECTED]> wrote:
>
>> Hi list,
>>
>> Are the any tools for parsing and extracting data from Hadoop's Job Logs?
>> I want to do stuff like ..
>>
>> 1. Getting run time of each map/reduce task
>> 2. Total map/reduce tasks ran on a particular node in that job  and some
>> similar stuff
>>
>> Any suggestions?
>>
>> Thanks
>>
>
>
+
anand sharma 2012-10-30, 02:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB