Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Debugging Mapreduce programs


Copy link to this message
-
RE: Debugging Mapreduce programs
Subroto Sanyal 2011-09-15, 05:18
Hi,

 

MapReduce framework provide different in built approaches to debug a Job:

1)       LocalJobRunner (Running the Job without a cluster; check basic
functionality of the Job implementation)

2)       IsolationRunner (Running a failed TaskAttempt on cluster- Remote
debugging can be done)

 

Apart from this we can use Sysouts and loggings in the Map or Reduce Tasks.

These logs will come in "sysout","syslog","syserr" accordingly. These files
will be present in the userlogs folder.

 

User can pass Remote Debug argument to Child Process to enable remote
debugging of Child process.

If you have access through putty, then you can use remote Debug option and
connect through eclipse as well.

 

 

Regards,
Subroto Sanyal

  _____  

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 15, 2011 9:49 AM
To: [EMAIL PROTECTED]
Subject: Re: Debugging Mapreduce programs

 

Shreya
There are multiple ways to debug map reduce programs. I found comfortable
using the jobtracker web UI. You can your job tracker web UI at
http://<server ip>:50030/jobtracker.jsp
Open it in a browser.
There you can find the running,failed and retired jobs. You can drill down
on any job to task tracker log levels from there.
Mostly I get the root cause of my job failures from there.

Hope it helps

Regards
Bejoy K S

  _____  

From: <[EMAIL PROTECTED]>

Date: Thu, 15 Sep 2011 09:39:23 +0530

To: <[EMAIL PROTECTED]>

ReplyTo: [EMAIL PROTECTED]

Subject: Debugging Mapreduce programs

 

Hi

 

Can we use sysouts in java to debug MR programs.

Where would those sysouts be printed?

 

What would be the best way to debug a remote system where I have access to
the machine only through putty.

 

Regards

Shreya

This e-mail and any files transmitted with it are for the sole use of the
intended recipient(s) and may contain confidential and privileged
information.
If you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on this
e-mail is strictly
prohibited and may be unlawful.