Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Hadoop 2.x with Eclipse


Copy link to this message
-
Re: Hadoop 2.x with Eclipse
Karim:
If you want to debug unit tests, using Eclipse is a viable approach.
Here is what I did the past week debugging certain part of hadoop
(JobSubmitter in particular) through an HBase unit test.

Run 'mvn install -DskipTests' to install hadoop locally
Open the class you want to debug and place breakpoint at proper location
Open unit test which depends on the class above and select Debug As ->
JUnit Test
When breakpoint hits, associate the sources.jar file in local maven repo
with the class. In my case, the sources jar file is located
under ~/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.1.2-SNAPSHOT

You should be able to step through hadoop code as usual at this point.

Cheers
On Sun, Oct 6, 2013 at 6:14 AM, John Lilley <[EMAIL PROTECTED]>wrote:

>  Karim,****
>
> ** **
>
> I am not an experienced Hadoop programmer, but what I found was that
> building and debugging Hadoop under Eclipse was very difficult, and I was
> never to make it work correctly.  I suggest using the well documented
> command-line Maven build, installing Hadoop from that build, and running it
> normally.  Once you have that working, run your namemode or datanode daemon
> so as to wait for a remote debugger attach before starting.  You should
> also get comfortable with log4j, the logging framework used by Hadoop, as
> those log files are often your best friend when trying to debug a
> collection of services.****
>
> ** **
>
> john****
>
> ** **
>
> *From:* Karim Awara [mailto:[EMAIL PROTECTED]]
> *Sent:* Sunday, October 06, 2013 5:41 AM
> *To:* user
> *Subject:* Hadoop 2.x with Eclipse****
>
> ** **
>
> Hi, ****
>
> I followed the instructions on how to import hadoop files to Eclipse (I am
> using hadoop 2.1 beta). ****
>
> Currently on my machine, I have hadoop 2.1 installed.. and its source code
> is imported on Eclipse. What I can't grasp is   how to proceed from there?
> ****
>
> I want to modify HDFS code (blockplacement strategy).. Now building hdfs
> project via generates errors to me (unresolved types in hadoop common).
> and if i built successfully, how to test my modified code?****
>
>
> ****
>
> --
> Best Regards,
> Karim Ahmed Awara****
>
> ** **
>  ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.****
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB