Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> How to understand Hadoop source code ?


+
Mohammad Mustaqeem 2013-04-17, 09:20
+
Arun C Murthy 2013-04-17, 22:06
+
Mohammad Mustaqeem 2013-04-18, 05:14
+
Prabakaran Krishnan 2013-04-18, 05:33
Copy link to this message
-
Re: How to understand Hadoop source code ?
Is there a specific bug fix or feature that you are trying to contribute?
 Specific questions like "how can I help with jira X?" or "what is the main
entry point when I run the hdfs command?" or "where does the namenode
serialize metadata to disk" or "where does the secondary namenode execute a
checkpoint" can help focus the conversation.

AFAIK, we don't have a general code walkthrough document focused on
onboarding new engineers.  This could be a valuable contribution if you
want to gather notes while you learn.  I think this always works best if
it's driven by a new engineer with review by an expert.  (If the experts
write it, then they might accidentally skip something non-obvious that
they've already internalized.)

Since that document doesn't exist yet, the other option is to do some
reading of the code, ideally while trying to fix a specific bug that has
been filed in jira.  Like you said, it's a relatively large codebase, so
it's impractical to read the whole thing top-to-bottom.  Instead, it's
important to look for high-level clues that steer you towards the right
files.  I've found that the Maven module structure and the Java package
names are usually descriptive enough to steer me in the right direction.
 If you focus on getting familiar with those, you'll basically build a
btree inside your brain that helps you index into the right part of the
codebase and answer your own questions rapidly.  Several examples:

"Where is the main entry point for the datanode daemon?": module
hadoop-hdfs, package org.apache.hadoop.hdfs.server.datanode

"What is the algorithm for rebalancing an unbalanced cluster?": module
hadoop-hdfs, package org.apache.hadoop.hdfs.server.balancer

"How does YARN launch a new container process?": module
hadoop-yarn-server-nodemanager,
package org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher

"Multiple daemons publish JMX metrics as a common concern.  Where is that
implemented?": module hadoop-common, package org.apache.hadoop.metrics2

I hope this is helpful to get the process started for you.  We're always
here to help if you have specific follow-up questions.

Thanks,
--Chris
On Wed, Apr 17, 2013 at 10:33 PM, Prabakaran Krishnan <
[EMAIL PROTECTED]> wrote:

> Couuld you please help me in understand map reduce in Hadoop?
>
>
>
> ________________________________
> From: Mohammad Mustaqeem <[EMAIL PROTECTED]>
> To: common-dev <[EMAIL PROTECTED]>
> Sent: Thursday, 18 April 2013 10:44 AM
> Subject: Re: How to understand Hadoop source code ?
>
>
> I am interested in HDFS. Please guide me.
>
>
> On Thu, Apr 18, 2013 at 3:36 AM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
>
> > Please don't cross post.
> >
> > What parts of Hadoop are you interested in? HDFS? YARN? MapReduce?
> >
> > Arun
> >
> > On Apr 17, 2013, at 2:50 PM, Mohammad Mustaqeem wrote:
> >
> > > Hello everyone,
> > >          I am new to this group. Since the source code of Hadoop is
> very
> > > big, I am not able to understand it entirely.
> > > Is there any document that describes the code?
> > > Is there any way to understand the functionality of each classes and
> its
> > > method?
> > >
> > >
> > > --
> > > *With regards ---*
> > > *Mohammad Mustaqeem*,
> > > M.Tech (CSE)
> > > MNNIT Allahabad
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
>
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
>
+
Noelle Jakusz 2013-04-18, 17:32
+
Steve Loughran 2013-04-18, 18:28
+
Noelle Jakusz 2013-04-19, 22:08
+
Mohammad Mustaqeem 2013-04-20, 06:08
+
Steve Loughran 2013-04-22, 15:43
+
Ronnie Ghose 2013-04-18, 17:47