Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Debug MR tasks impossible.


Copy link to this message
-
Re: Debug MR tasks impossible.
You can run a job in local mode without a cluster and run in a debugger
enabling breakpoints single stepping - examination of variables ...
If you do this
1)  Choose a SMALL problem
2) Beware of the fact that there is only one VM - that means be cautions
with static variables - they cause leakage from mappers to reducers.
3) Keep testing on the cluster
My rule is never try a task on a cluster until it works well in local mode

On Thu, Mar 29, 2012 at 9:13 AM, Wydra, Paul <[EMAIL PROTECTED]> wrote:

> You can also decompose your problem so that you have code that is testable
> outside the map-reduce process.  This way, you can ensure all your "helper"
> classes/methods are behaving as you expect, and you can debug this locally.
>  Once that is done, you are left with only debugging things related to MR
> plumbing.  MRUnit is good for this as Brock pointed out.  But even if you
> can't use that for some reason, knowing the helper logic is correct is a
> big help with complex jobs.
>
> -paul
>
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, March 29, 2012 12:08 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Debug MR tasks impossible.
>
> Can I suggest unit testing with MRUnit:
>
> https://cwiki.apache.org/confluence/display/MRUNIT/Index
> https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial
>
> Brock
>
> On Thu, Mar 29, 2012 at 10:33 AM, Pedro Costa <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I'm trying to debug map and reduce tasks for a quite long time, and it
> seems
> > that it's impossible. MR are launched in new process and there's no way
> to
> > debug them. Even with IsolationRunner class it's impossible. This isn't
> good
> > because I really need to debug the class, to understand some changes
> that I
> > made to the code.
> >
> > I wonder how MapReduce programmers could debug the code that they
> > implemented?
> >
> > --
> > Best regards,
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com