Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - Re: ReplicationTargetChooser has incorrect block placement comments


+
Harsh J 2013-04-04, 06:14
+
Harsh J 2013-04-04, 07:07
Copy link to this message
-
Re: ReplicationTargetChooser has incorrect block placement comments
Harsh J 2013-04-04, 08:58
Hi Mei,

Thanks. Please also keep [EMAIL PROTECTED] on the to: list
when replying, so your questions get exposure from other HDFS devs as
well :)

Regarding your question, the answer is no, the NN will never violate
the block placement policy on its own when choosing block location
targets for an allocation.

On Thu, Apr 4, 2013 at 2:22 PM, Mei Long <[EMAIL PROTECTED]> wrote:
> I will open a doc JIRA. Another question: is there a chance that all three
> blocks end up on the same rack of no DN exists on the writer node? Or is the
> NN knows to put the other two copies on a different rack?
>
>
> On Thu, Apr 4, 2013 at 3:07 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Good to know! Please also consider improving the javadoc on that
>> method to reflect this, by logging a doc JIRA under HDFS :)
>>
>> On Thu, Apr 4, 2013 at 12:33 PM, Mei Long <[EMAIL PROTECTED]> wrote:
>> > Harsh,
>> >
>> > Thank you! This is exactly what I was confused about. By looking at the
>> > code
>> > I was suspecting that it's not looking at rack locality for the second
>> > scenario (No DN on writer node.) But I was unable to confirm it without
>> > running the code.
>> >
>> > Your answer makes so much sense. Thank you!! :-) I'm much happier now.
>> >
>> > On Thu, Apr 4, 2013 at 2:14 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi Mei,
>> >>
>> >> These questions are best fit for the whole hdfs-dev@ group, which am
>> >> adding while responding back.
>> >>
>> >> You're correct that the local node being not chosen, a random node on
>> >> the same rack as the writer may be tried. However, if the writer node
>> >> itself had no local DN in the first place, a completely random node
>> >> across any rack may be selected.
>> >>
>> >> This is visible at
>> >>
>> >>
>> >> http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java?view=markup
>> >> (~263).
>> >>
>> >> On Thu, Apr 4, 2013 at 12:53 AM, Mei Long <[EMAIL PROTECTED]> wrote:
>> >> > Hi Harsh,
>> >> >
>> >> > I found your post on JIRA regarding the comments. I find them very
>> >> > confusing
>> >> > as well. I'm trying to figure out the following comment:
>> >> >
>> >> >  * the 1st replica is placed on the local machine,
>> >> >  * otherwise a random datanode.
>> >> >
>> >> > Do you know when it says "otherwise a random datanode," does it mean
>> >> > any
>> >> > random datanode anywhere on the network or a random datanode on the
>> >> > same
>> >> > rack as the local machine? I've been looking at the code for an hour
>> >> > and
>> >> > I'm
>> >> > getting more confused with the comment and code in chooseLocalNode()
>> >> >
>> >> >   /* choose <i>localMachine</i> as the target.
>> >> >
>> >> >    * if <i>localMachine</i> is not availabe,
>> >> >
>> >> >    * choose a node on the same rack
>> >> >
>> >> >    * @return the choosen node
>> >> >
>> >> >    */
>> >> >
>> >> > Your help is much appreciated!
>> >> >
>> >> > Mei
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>

--
Harsh J