Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> About debugging HDFS and MapReduce...


Copy link to this message
-
Re: About debugging HDFS and MapReduce...
I am not sure what you mean by some portion of source code of Hadoop.
Because you are running on a fully distributed environment you need to
make sure that eclipse is attached to the correct process.  If you are
trying to debug the data upload pipeline then you need to be sure that
eclipse is attached to the proper data node.  It is hard to guess which
one because you setup has 10 to choose from.  Your image showed you
attached to the client, you have to use remote debugging to debug a
different process.

The next thing you are likely to run into is timeout issues.  Almost all
of the code has a lot of timeouts, to be able to detect when another
process has stopped responding.  This is a bigger problem in MR where you
will be walking through code and suddenly the process disappears because
it was shot be the Task Tracker or Node Manager.  I am not as familiar
with the HDFS timeouts, but you probably want to be aware that they exist.

--Bobby

On 10/7/13 12:38 AM, "Yoonmin Nam" <[EMAIL PROTECTED]> wrote:

>Sorry for this inconvenience.
>I uploaded image file on our lab server.
>
>http://infolab.dgist.ac.kr/~ronymin/pictures/debug.png
>
>-----Original Message-----
>From: haosdent [mailto:[EMAIL PROTECTED]]
>Sent: Monday, October 07, 2013 2:24 PM
>To: [EMAIL PROTECTED]; 남윤민
>Subject: Re: About debugging HDFS and MapReduce...
>
>I still couldn't see the image. The page shows that "The page you
>requested could not be found.".
>
>--
>Best Regards,
>Haosong Huang
>Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
>On Monday, October 7, 2013 at 12:38 PM, 남윤민 wrote:
>
>> Here is image I uploaded.
>>  
>> Sorry for that!
>>  
>> http://mail.dgist.ac.kr/design/common/image/attaches//ronymin_167_inli
>> ne0_6806441608437847.png
>> > --- Original Message ---
>> > From : "Ted Yu"<[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> To
>> > : [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
>> > Cc : [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
>> > Date : 2013/10/07 월요일 오후 1:22:45
>> > Subject : Re: About debugging HDFS and MapReduce...
>> >  
>> > Your image didn't go through.
>> > You may want to upload it to image sharing site.
>> >  
>> > What version of Hadoop are you using ?
>> >  
>> > Thanks
>> >  
>> > On Oct 6, 2013, at 8:56 PM, 남윤민 <[EMAIL PROTECTED]
>>(mailto:[EMAIL PROTECTED])> wrote:
>> > > Hello!!
>> > > My name is Yoonmin Nam from South Korea.
>> > >
>> > > Currently, I am trying to debug namenode and datanode using
>> > > eclipse and fully-distributed environemnt in my lab. (1 master and
>> > > 10 nodes)
>> > >
>> > > However, I can do debug but not fully, just some portion of source
>>code of hadoop.
>> > >
>> > > This means I cannot see how some daemon thread such as DataStreamer.
>> > > Also, I want to see how NameNode thread and DataNode thread work
>>but I can't.
>> > >
>> > > As you can see, there is Daemon thread (Thre ad-2) and it might be
>>DataStreamer thread but.. I can't see.
>> > >
>> > >
>> > >
>> > >
>> > > So, I think everyone in the hdfs-dev have very professional
>>experience in developing HDFS and MapReduce and I want to ask you about
>>how can I see all real daemons are working for academic reason.
>> > >
>> > > Even you have no idea about it, may I ask you that how you test
>>your implementation of HDFS or MapReduce?
>> > >
>> > >
>> > > Thanks!!
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > // Yoonmin Nam
>> > >
>>  
>>  
>> // Yoonmin Nam
>>  
>>  
>
>
>
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB