Hi,

When I run a MapReduce job, the userlogs are stored on each Datanode.

Like: ~/hadoop_drive/hadoop/mapred/local/userlogs/job_201211011005_0018

Under this directory I have on subdirectory per job, like:
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000010_0
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000016_0
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000017_0
drwx------ 2 hadoop hadoop 4096 nov  5 12:46
attempt_201211011005_0018_m_000021_0

Now, I want to take a look at all the logs. Is there an easy way to
retrieve them? Or do I need to log in each single node and look at
each directory of each task?

I would like to put all the logs into a single file.

Thanks,

JM

Just watch logs in jobtracker & history UI isn't enough for you?
On Thu, Nov 8, 2012 at 9:44 AM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> When I run a MapReduce job, the userlogs are stored on each Datanode.
>
> Like: ~/hadoop_drive/hadoop/mapred/local/userlogs/job_201211011005_0018
>
> Under this directory I have on subdirectory per job, like:
> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
> attempt_201211011005_0018_m_000010_0
> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
> attempt_201211011005_0018_m_000016_0
> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
> attempt_201211011005_0018_m_000017_0
> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
> attempt_201211011005_0018_m_000021_0
>
> Now, I want to take a look at all the logs. Is there an easy way to
> retrieve them? Or do I need to log in each single node and look at
> each directory of each task?
>
> I would like to put all the logs into a single file.
>
> Thanks,
>
> JM
>

Do you mean under http://servername:50030/jobhistoryhome.jsp ?

>From there I can see the local logs, or the job results/summary. But I
can't see the logs of the other nodes. If I want to see the logs, I
need to click on the job ID, then on the map successful tasks, then on
all the tasks one by one, then on view all log. Ouf. It's long.

On my MapReduce I'm doing that:
Logger.getLogger(TimestampBasedRowMover.class).info ("Moving " +
Bytes.toString(Bytes.tail(values.getRow(), values.getRow().length -
8)));

That's what I'm trying to see on my logs :(

2012/11/7, Binglin Chang <[EMAIL PROTECTED]>:
> Just watch logs in jobtracker & history UI isn't enough for you?
>
>
> On Thu, Nov 8, 2012 at 9:44 AM, Jean-Marc Spaggiari
> <[EMAIL PROTECTED]
>> wrote:
>
>> Hi,
>>
>> When I run a MapReduce job, the userlogs are stored on each Datanode.
>>
>> Like: ~/hadoop_drive/hadoop/mapred/local/userlogs/job_201211011005_0018
>>
>> Under this directory I have on subdirectory per job, like:
>> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
>> attempt_201211011005_0018_m_000010_0
>> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
>> attempt_201211011005_0018_m_000016_0
>> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
>> attempt_201211011005_0018_m_000017_0
>> drwx------ 2 hadoop hadoop 4096 nov  5 12:46
>> attempt_201211011005_0018_m_000021_0
>>
>> Now, I want to take a look at all the logs. Is there an easy way to
>> retrieve them? Or do I need to log in each single node and look at
>> each directory of each task?
>>
>> I would like to put all the logs into a single file.
>>
>> Thanks,
>>
>> JM
>>
>