Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> MapReduce: Read the logs


Copy link to this message
-
MapReduce: Read the logs
Hi,

When I run a MapReduce job, the userlogs are stored on each Datanode.

Like: ~/hadoop_drive/hadoop/mapred/local/userlogs/job_201211011005_0018

Under this directory I have on subdirectory per job, like:
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000010_0
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000016_0
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000017_0
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000021_0

Now, I want to take a look at all the logs. Is there an easy way to
retrieve them? Or do I need to log in each single node and look at
each directory of each task?

I would like to put all the logs into a single file.

Thanks,

JM
+
Binglin Chang 2012-11-08, 01:57
+
Jean-Marc Spaggiari 2012-11-08, 02:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB