Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> modify HDFS


Copy link to this message
-
Re: modify HDFS
Karim!

Hadoop 3.0 corresponds to trunk currently. I would recommend you to use branch-2. Its fairly stable. hadoop-1.x is rather old and is in maintenance mode now. You can get all the branches from https://wiki.apache.org/hadoop/GitAndHadoop

git clone git://git.apache.org/hadoop-common.git
Please perusehttps://hadoop.apache.org/docs/stable/cluster_setup.html to configure a distributed cluster. The block files should be in dfs.data.dir on the data nodes. You may also be able to use the Offline Image viewer to find all blocks and their locations from the fsimage file.

HTH
Ravi
________________________________
 From: Pradeep Gollakota <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, October 2, 2013 9:29 AM
Subject: Re: modify HDFS
 
Since hadoop 3.0 is 2 major versions higher, it will be significantly different than working with hadoop 1.1.2. The hadoop-1.1 branch is available on SVN at http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1/

On Tue, Oct 1, 2013 at 11:30 PM, Karim Awara <[EMAIL PROTECTED]> wrote:

Hi all,
>
>
>My previous web surfing led me to such steps that I executed successfully. However, my issue is that what version of hadoop this is? (I believe it is hadoop 3.0 since it supports maven build). However, I want to modify a stable release (hadoop 1.1.2, which does not have the maven build support).  Will working with the hadoop 3.0 make a lot of difference to me?
>
>
>Secondly, my goal is to modify the block placement strategy at HDFS in a distributed environment, and test such changes myself. Now, assuming I was successful in modifying the HDFS code, how to test such modifications (since the hadoop version is actually missing configuration files and so on that make it work in distributed environment)?
>
>
>
>--
>Best Regards,
>Karim Ahmed Awara
>
>
>On Wed, Oct 2, 2013 at 1:13 AM, Ravi Prakash <[EMAIL PROTECTED]> wrote:
>
>Karim!
>>
>>
>>You should read BUILDING.txt . I usually generate the eclipse files using
>>
>>
>>
>>mvn eclipse:eclipse
>>
>>
>>Then I can import all the projects into eclipse as eclipse projects. This is useful for code navigation and completion etc. however I still build using command line:
>>mvn -Pdist -Dmaven.javadoc.skip -DskipTests install
>>
>>HTH
>>Ravi
>>
>>
>>
>>
>>________________________________
>> From: Jagat Singh <[EMAIL PROTECTED]>
>>To: [EMAIL PROTECTED]
>>Sent: Tuesday, October 1, 2013 3:44 PM
>>Subject: Re: modify HDFS
>>
>>
>>
>>Hi,
>>What issue you having.
>>I Wrote it about here , might help you
>>Import it into eclipse as maven project
>>http://jugnu-life.blogspot.com.au/2013/09/build-and-compile-hadoop-from-source.html?m=1
>>Thanks
>>On 01/10/2013 11:56 PM, "Karim Awara" <[EMAIL PROTECTED]> wrote:
>>
>>Hi,
>>>
>>>I want to modify the source code of HDFS to my usage, but I can't get any handy sources for development of hdfs on eclipse. (I tried the hdfs developer mailing list, but they are unresponsive). May you guide me?
>>>
>>>
>>>
>>>--
>>>Best Regards,
>>>Karim Ahmed Awara
>>>
>>>
>>>________________________________
>>>This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
>>
>>
>
>
>
>
>________________________________
>This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.