-Re: Debug MR tasks impossible.
Steve Lewis 2012-03-29, 16:32
You can run a job in local mode without a cluster and run in a debugger
enabling breakpoints single stepping - examination of variables ...
If you do this
1) Choose a SMALL problem
2) Beware of the fact that there is only one VM - that means be cautions
with static variables - they cause leakage from mappers to reducers.
3) Keep testing on the cluster
My rule is never try a task on a cluster until it works well in local mode
On Thu, Mar 29, 2012 at 9:13 AM, Wydra, Paul <[EMAIL PROTECTED]> wrote:
> You can also decompose your problem so that you have code that is testable
> outside the map-reduce process. This way, you can ensure all your "helper"
> classes/methods are behaving as you expect, and you can debug this locally.
> Once that is done, you are left with only debugging things related to MR
> plumbing. MRUnit is good for this as Brock pointed out. But even if you
> can't use that for some reason, knowing the helper logic is correct is a
> big help with complex jobs.
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, March 29, 2012 12:08 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Debug MR tasks impossible.
> Can I suggest unit testing with MRUnit:
> On Thu, Mar 29, 2012 at 10:33 AM, Pedro Costa <[EMAIL PROTECTED]> wrote:
> > Hi,
> > I'm trying to debug map and reduce tasks for a quite long time, and it
> > that it's impossible. MR are launched in new process and there's no way
> > debug them. Even with IsolationRunner class it's impossible. This isn't
> > because I really need to debug the class, to understand some changes
> that I
> > made to the code.
> > I wonder how MapReduce programmers could debug the code that they
> > implemented?
> > --
> > Best regards,
> Apache MRUnit - Unit testing MapReduce -
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033