Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - MapReduce logs


Copy link to this message
-
Re: MapReduce logs
Jean-Marc Spaggiari 2012-11-21, 19:55
Thanks for the info.

I have quickly draft this bash script in case it can help someone...
You just neeed to make sure the IP inside is replaced.
To call it, you need to give the job task page.

./showLogs.sh "http://192.168.23.7:50030/jobtasks.jsp?jobid=job_201211211408_0001&type=map&pagenum=1"

Then you can redirect the output, or do what ever you want.

I was wondering if there was a "nicer" solution...

:~/test$ cat showLogs.sh
#!/bin/bash
rm -f tasks.html
wget --quiet --output-document tasks.html $1
for i in `cat tasks.html | grep taskdetails | cut -d"\"" -f2 | grep
taskdetails`; do
rm -f tasksdetails.html
wget --quiet --output-document tasksdetails.html http://192.168.23.7:50030/$i
for j in `cat tasksdetails.html | grep "all=true" | cut -d"\"" -f6`; do
printf "*"%.0s {1..80}
echo
echo $j
printf "*"%.0s {1..80}
echo
rm -f logs.txt
wget --quiet --output-document logs.txt $j
tail -n +31 logs.txt | head -n -2
done
done
rm -f tasks.html
rm -f tasksdetails.html
rm -f logs.txt
2012/11/21, Dino Kečo <[EMAIL PROTECTED]>:
> Hi,
>
> We had similar requirement and we built small Java application which gets
> information about task nodes from Job Tracker and download logs into one
> file using URLs of each task tracker.
>
> For huge logs this becomes slow and time consuming.
>
> Hope this helps.
>
> Regards,
> Dino Kečo
> msn: [EMAIL PROTECTED]
> mail: [EMAIL PROTECTED]
> skype: dino.keco
> phone: +387 61 507 851
>
>
> On Wed, Nov 21, 2012 at 7:55 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> When we run a MapReduce job, the logs are stored on all the tasktracker
>> nodes.
>>
>> Is there an easy way to agregate all those logs together and see them
>> in a single place instead of going to the tasks one by one and open
>> the file?
>>
>> Thanks,
>>
>> JM
>>
>