Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: good way to debug map reduce code


Copy link to this message
-
Re: good way to debug map reduce code
For Java MR jobs, there is Apache MRUnit that provides a good way of
writing test cases. See http://mrunit.apache.org

On Wed, Dec 26, 2012 at 7:26 AM, jamal sasha <[EMAIL PROTECTED]> wrote:
> Hi,
>   I have been using python hadoop streaming framework to write the code and
> now I am slowly moving towards the core java api's.
> And I am getting comfortable with it but what is the quickest way to debug
> the map reduce native code..
> like in hadoop streaming this worked great.
> % cat input.txt | python mapper.py | sort | python reducer.py
>
> If there use to be any coding error.. it use to just throw them off and it
> was very fast to debug as you code.
> Is there any similar way .. where i dont have to run hadoop jobs to debg and
> wait and go thru hadoop logs to see that maybe i miss a semi-colon..
> Thanks
> Jamal

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB