Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Issues during setting up hadoop security cluster


Copy link to this message
-
Re: Issues during setting up hadoop security cluster
Vinod Kumar Vavilapalli 2012-01-20, 05:23
Hi,

Just today evening, I happened to run into someone who had the same
issue. After some debugging, I cornered that to the hostnames having
upper-case characters. Somehow, when DataNode or NodeManager try to
get a service ticket for their corresponding services (NameNode and
ResourceManager respectively), the hostname were getting converted
into all lowercase. You can see if it is the same situation with you
by looking at krb5kdc logs.

If that is the case, changing the hostnames everywhere to be all
small-case may help. Please try that and let me know.

HTH,
+Vinod
On Thu, Jan 19, 2012 at 8:52 PM, Emma Lin <[EMAIL PROTECTED]> wrote:
> Gurus,
>
> I’m setting up a security cluster of hadoop .23. But now, the communication
> between Data Node and Name Node, Node Manager and Resource Manager have
> problem.
>
> When I start the Node Manager, it will report following error, and then
> shutdown itself. Did you ever see such issue? Do you have any idea on how to
> triage this issue?
>
>
>
> 2012-01-20 12:03:08,258 INFO  ipc.HadoopYarnRPC
> (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker
>
> 2012-01-20 12:03:08,291 INFO  nodemanager.NodeStatusUpdaterImpl
> (NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to
> ResourceManager at hadoopRM.example.aurora:9003
>
> 2012-01-20 12:03:20,399 WARN  ipc.Client (Client.java:run(526)) - Couldn't
> setup connection for nm/[EMAIL PROTECTED]RA to
> rm/[EMAIL PROTECTED]RA
>
> 2012-01-20 12:03:20,405 ERROR service.CompositeService
> (CompositeService.java:start(72)) - Error starting services
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>
> org.apache.avro.AvroRuntimeException:
> java.lang.reflect.UndeclaredThrowableException
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132)
>
>         at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231)
>
> Caused by: java.lang.reflect.UndeclaredThrowableException
>
>         at
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:161)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128)
>
>         ... 3 more
>
> Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed
> on local exception: java.io.IOException: Couldn't setup connection for
> nm/[EMAIL PROTECTED]RA to
> rm/[EMAIL PROTECTED]RA; Host Details : local host is:
> "hadoopNM/10.112.127.102"; destination host is:
> ""hadoopRM.example.aurora":9003;
>
>         at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139)
>
>         at $Proxy14.registerNodeManager(Unknown Source)
>
>         at
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
>
>         ... 5 more
>
> Caused by: java.io.IOException: Failed on local exception:
> java.io.IOException: Couldn't setup connection for
> nm/[EMAIL PROTECTED]RA to
> rm/[EMAIL PROTECTED]RA; Host Details : local host is:
> "hadoopNM/10.112.127.102"; destination host is:
> ""hadoopRM.example.aurora":9003;
>
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1089)
>
>         at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136)