Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> How do I log from my map/reduce application?


Copy link to this message
-
Re: How do I log from my map/reduce application?
W. P.,

How are you running your Reducer? Is everything running in standalone mode
(all mappers/reducers in the same process as the launching application)? Or
are you running this in pseudo-distributed mode or on a remote cluster?

Depending on the application's configuration, log4j configuration could be
read from one of many different places.

Furthermore, where are you expecting your output? If you're running in
pseudo-distributed (or fully distributed) mode, mapper / reducer tasks will
not emit output back to the console of the launching application.  That only
happens in local mode. In the distributed flavors, you'll see a different
file for each task attempt containing its log output, on the machine where
the task executed. These files can be accessed through the web UI at
http://jobtracker:50030/ -- click on the job, then the task, then the task
attempt, then "syslog" in the right-most column.

- Aaron

On Mon, Dec 13, 2010 at 10:05 AM, W.P. McNeill <[EMAIL PROTECTED]> wrote:

> I would like to use Hadoop's Log4j infrastructure to do logging from my
> map/reduce application.  I think I've got everything set up correctly, but
> I
> am still unable to specify the logging level I want.
>
> By default Hadoop is set up to log at level INFO.  The first line of its
> log4j.properties file looks like this:
>
> hadoop.root.logger=INFO,console
>
>
> I have an application whose reducer looks like this:
>
> package com.me;
>
> public class MyReducer<...> extends Reducer<...> {
>   private static Logger logger > Logger.getLogger(MyReducer.class.getName());
>
>   ...
>   protected void reduce(...) {
>       logger.debug("My message");
>       ...
>   }
> }
>
>
> I've added the following line to the Hadoop log4j.properties file:
>
> log4j.logger.com.me.MyReducer=DEBUG
>
>
> I expect the Hadoop system to log at level INFO, but my application to log
> at level DEBUG, so that I see "My message" in the logs for the reducer
> task.
>  However, my application does not produce any log4j output.  If I change
> the
> line in my reducer to read logger.info("My message") the message does get
> logged, so somehow I'm failing to specify that log level for this class.
>
> I've also tried changing the log4j line for my app to
> read log4j.logger.com.me.MyReducer=DEBUG,console and get the same result.
>
> I've been through the Hadoop and log4j documentation and I can't figure out
> what I'm doing wrong.  Any suggestions?
>
> Thanks.
>