Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> Datanode registration, port number


+
Dhaivat Pandya 2013-12-23, 04:11
+
Shekhar Sharma 2013-12-23, 04:28
+
Dhaivat Pandya 2013-12-23, 04:38
Copy link to this message
-
Re: Datanode registration, port number
Any other ideas?
On Sun, Dec 22, 2013 at 10:38 PM, Dhaivat Pandya <[EMAIL PROTECTED]>wrote:

> I understand that that is how the port is *later* retrieved, but how does
> the namenode know the port in the first place? i.e. if the datanode sends a
> packet to the namenode, how does the namenode know what port # the datanode
> server is running on?
>
>
> On Sun, Dec 22, 2013 at 10:28 PM, Shekhar Sharma <[EMAIL PROTECTED]>wrote:
>
>> Check the version file
>> During the startup Datanode sends out a heart beat signal and NN
>> assigns a storageID for it which is unique for every datanode.
>> Using the storage id it identify the ip and port of a datanode
>> Regards,
>> Som Shekhar Sharma
>> +91-8197243810
>>
>>
>> On Mon, Dec 23, 2013 at 9:41 AM, Dhaivat Pandya <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > I'm currently trying to build a cache layer that should sit "on top" of
>> the
>> > datanode. Essentially, the namenode should know the port number of the
>> > cache layer instead of that of the datanode (since the namenode then
>> relays
>> > this information to the default HDFS client). All of the communication
>> > between the datanode and the namenode currently flows through my cache
>> > layer (including heartbeats, etc.)
>> >
>> > *First question*: is there a way to tell the namenode where a datanode
>> > should be? Any way to trick it into thinking that the datanode is on a
>> port
>> > number where it actually isn't? As far as I can tell, the port number is
>> > obtained from the DatanodeId object; can this be set in the
>> configuration
>> > so that the port number derived is that of the cache layer?
>> >
>> > I spent quite a bit of time on the above question and I could not find
>> any
>> > sort of configuration option that would let me do that. So, I delved
>> into
>> > the HDFS source code and tracked down the DatanodeRegistration class.
>> > However, I can't seem to find out *how* the NameNode figures out the
>> > Datanode's port number or if I could somehow change the packets to
>> reflect
>> > the port number of cache layer? *Second question: *how does the namenode
>> > figure out a newly-registered Datanode's port number?
>> >
>> > Thank you,
>> >
>> > Dhaivat
>>
>
>
+
Harsh J 2013-12-24, 05:46
+
Dhaivat Pandya 2013-12-24, 05:57
+
Andrew Wang 2013-12-30, 20:48
+
Dhaivat Pandya 2013-12-31, 20:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB