A quick question about the user logs (hadoop 0.19.1).

I've looked all over HDFS.  I can't locate the actual userlogs.  Where are they stored?  I can get to them through the job tracker web interface, but I'd like to be able to script some jobs to check the logs for errors.  I suppose I could use curl against the web interface, but that seems like a hack.  Alternatively, is there command line access to the userlogs through the hadoop command?  I couldn't find anything like that.
Thanks for any tips...

Marc
Feeva Technology

________________________________
PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

Hi Marc,
you can find the userlogs under the directory
<your-hadoop-dir>/logs/userlogs

for example hadoop-0.19.0/logs/userlogs

On Wed, Jul 29, 2009 at 5:36 PM, Marc Limotte <[EMAIL PROTECTED]> wrote:

> A quick question about the user logs (hadoop 0.19.1).
>
> I've looked all over HDFS.  I can't locate the actual userlogs.  Where are
> they stored?  I can get to them through the job tracker web interface, but
> I'd like to be able to script some jobs to check the logs for errors.  I
> suppose I could use curl against the web interface, but that seems like a
> hack.  Alternatively, is there command line access to the userlogs through
> the hadoop command?  I couldn't find anything like that.
> Thanks for any tips...
>
> Marc
> Feeva Technology
>
> ________________________________
> PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR
> ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION
> PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE,
> DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY
> PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND
> PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
>

On Wed, Jul 29, 2009 at 10:06 PM, Marc Limotte <[EMAIL PROTECTED]> wrote:

> A quick question about the user logs (hadoop 0.19.1).
>
> I've looked all over HDFS.  I can't locate the actual userlogs.  Where are
> they stored?  I can get to them through the job tracker web interface, but
> I'd like to be able to script some jobs to check the logs for errors.  I
> suppose I could use curl against the web interface, but that seems like a
> hack.  Alternatively, is there command line access to the userlogs through
> the hadoop command?  I couldn't find anything like that.
> Thanks for any tips...
>
>
All hadoop logs are stored on the local filesystem on each node (the
location is configurable). What you could do though is run a cron job which
periodically pushes log files to HDFS at the end of the day.

Here's another idea:
http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-hadoop-log-collection/

--
Harish Mallipeddi
http://blog.poundbang.in

Ah, thanks for the info.  I didn't think to look locally.

-----Original Message-----
From: bonito perdo [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, July 29, 2009 9:41 AM
To: [EMAIL PROTECTED]
Subject: Re: userlogs

Hi Marc,
you can find the userlogs under the directory
<your-hadoop-dir>/logs/userlogs

for example hadoop-0.19.0/logs/userlogs

On Wed, Jul 29, 2009 at 5:36 PM, Marc Limotte <[EMAIL PROTECTED]> wrote:

> A quick question about the user logs (hadoop 0.19.1).
>
> I've looked all over HDFS.  I can't locate the actual userlogs.  Where are
> they stored?  I can get to them through the job tracker web interface, but
> I'd like to be able to script some jobs to check the logs for errors.  I
> suppose I could use curl against the web interface, but that seems like a
> hack.  Alternatively, is there command line access to the userlogs through
> the hadoop command?  I couldn't find anything like that.
> Thanks for any tips...
>
> Marc
> Feeva Technology
>
> ________________________________
> PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR
> ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION
> PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE,
> DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY
> PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND
> PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
>

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.