Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Datanode registration, port number


Copy link to this message
-
Re: Datanode registration, port number
Dhaivat Pandya 2013-12-24, 02:24
Any other ideas?
On Sun, Dec 22, 2013 at 10:38 PM, Dhaivat Pandya <[EMAIL PROTECTED]>wrote:

> I understand that that is how the port is *later* retrieved, but how does
> the namenode know the port in the first place? i.e. if the datanode sends a
> packet to the namenode, how does the namenode know what port # the datanode
> server is running on?
>
>
> On Sun, Dec 22, 2013 at 10:28 PM, Shekhar Sharma <[EMAIL PROTECTED]>wrote:
>
>> Check the version file
>> During the startup Datanode sends out a heart beat signal and NN
>> assigns a storageID for it which is unique for every datanode.
>> Using the storage id it identify the ip and port of a datanode
>> Regards,
>> Som Shekhar Sharma
>> +91-8197243810
>>
>>
>> On Mon, Dec 23, 2013 at 9:41 AM, Dhaivat Pandya <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > I'm currently trying to build a cache layer that should sit "on top" of
>> the
>> > datanode. Essentially, the namenode should know the port number of the
>> > cache layer instead of that of the datanode (since the namenode then
>> relays
>> > this information to the default HDFS client). All of the communication
>> > between the datanode and the namenode currently flows through my cache
>> > layer (including heartbeats, etc.)
>> >
>> > *First question*: is there a way to tell the namenode where a datanode
>> > should be? Any way to trick it into thinking that the datanode is on a
>> port
>> > number where it actually isn't? As far as I can tell, the port number is
>> > obtained from the DatanodeId object; can this be set in the
>> configuration
>> > so that the port number derived is that of the cache layer?
>> >
>> > I spent quite a bit of time on the above question and I could not find
>> any
>> > sort of configuration option that would let me do that. So, I delved
>> into
>> > the HDFS source code and tracked down the DatanodeRegistration class.
>> > However, I can't seem to find out *how* the NameNode figures out the
>> > Datanode's port number or if I could somehow change the packets to
>> reflect
>> > the port number of cache layer? *Second question: *how does the namenode
>> > figure out a newly-registered Datanode's port number?
>> >
>> > Thank you,
>> >
>> > Dhaivat
>>
>
>