Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - NameNode uses up (almost) all ports


Copy link to this message
-
Re: NameNode uses up (almost) all ports
Robert Molina 2012-10-31, 18:41
Hi James,
Since it seems intermittent, have you verified if there are any maintenance
type procedures being done within the namenode machine or it's related
network?

On Sun, Oct 28, 2012 at 9:51 PM, Jianhui Zhang <[EMAIL PROTECTED]>wrote:

> Hi folks,
>
> We've got this weird problem regularly on our NameNode (apache
> hadoop-0.20.205.0) - every couple of weeks:
>
> The JobTracker had this error:
>
> 2012-10-08 11:44:03,928 WARN org.apache.hadoop.hdfs.DFSClient: Problem
> renewing lease for DFSClient_1416124356
> java.io.IOException: Call to nn-virtual.x.y.z/1.2.3.4:8020 failed on
> local exception: java.net.BindException: Cannot assign requested address
>         at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1071)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>         at $Proxy5.renewLease(Unknown Source)
>
> and
>
> 2012-10-08 11:44:03,927 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: nn-virtual.x.y.z/1.2.3.4:8020. Already tried 9 time(s).
>
> in which "nn-virtual.x.y.z/1.2.3.4:8020" is our HDFS address.
>
> I listed out all the local addresses on the NN and got about 24K (more or
> less) open ports. The ip_local_port_range has:
>
> 32768 61000
>
> We are not reaching the limit, but very close. What's strange is: almost
> all of the local ports are used by the NN process. There might be some
> holes in the list, but overall, it seems the NN was using up all the
> ephemeral ports available in the range.
>
> Right now, I strongly suspect that "Cannot assign requested address" is
> due to lack of ports - although I'm not 100% sure since the ephemeral ports
> change all the time.
>
> Has anybody seen this before?  Any pointers would be appreciated.
>
> Also, we are using a virtual IP for the NN. All the ports are opened on
> the virtual IP address. Could it be related to the problem?
>
>  Thanks for your help,
> James
>