Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> 'Can't get service ticket for: host/0.0.0.0' when running hdfs with kerberos


Copy link to this message
-
Re: 'Can't get service ticket for: host/0.0.0.0' when running hdfs with kerberos
Is your default kerberos realm set to "EXAMPLETEST.COM<http://EXAMPLETEST.COM>"?  If not, have you tried grepping your confs for "EXAMPLETEST.COM<http://EXAMPLETEST.COM>"?

Daryn

On Sep 12, 2012, at 5:37 PM, jack chrispoo wrote:

Hi,

I'm using Hadoop 1.0.1, I tried to follow https://ccp.cloudera.com/display/CDHDOC/Configuring+Hadoop+Security+in+CDH3+%28KSSL%29 to configure hadoop with kerberos authentication. I configured KDC and added hdfs, mapred, host principles for each node to kerberos and deployed the keytabs to each node.

I modified core-site.xml, hdfs-site.xml, hadoop-env.sh as below, and then tried to start dfs using sudo hadoop_dir/bin/start-dfs.sh

The NameNode and DataNodes started without error. And from namenode:50070 I can see that all DataNodes are live. I can create directories, ls in hdfs using hadoop command. But one thing I'm confused is: earlier when I started hdfs without Kerberos, 'jps' will show in namenode a pid with 'NameNode':

  3239 NameNode

and in datanode a pid with 'DataNode':

  24307 DataNode

, but now 'jps' shows a pid with 'NameNode' on namenode,

  3239 NameNode

but a pid without any name on DataNode,

  # jps
  2931 Jps
  2684

I guess this process 2684 is the DataNode because if I run 'sudo hadoop_dir/bin/stop-dfs.sh' this process goes away. Has anyone seen this before? why it doesn't show 'DataNode'?

Also, a while after I started hdfs, NameNode's log showed some error:

2012-09-12 14:31:06,335 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:host/[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> cause:java.io.IOException: Can't get service ticket for: host/0.0.0.0<http://0.0.0.0/>
2012-09-12 14:31:06,335 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:host/node0@ EXAMPLETEST.COM<http://EXAMPLETEST.COM/> cause:java.io.IOException: Can't get service ticket for: host/0.0.0.0<http://0.0.0.0/>
2012-09-12 14:31:06,358 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.io.IOException: Can't get service ticket for: host/0.0.0.0<http://0.0.0.0/>
        at org.apache.hadoop.security.SecurityUtil.fetchServiceTicket(SecurityUtil.java:138)
        at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:158)
        at org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1$1.run(GetImageServlet.java:88)
        at org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1$1.run(GetImageServlet.java:85)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:85)
        at org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:70)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.hadoop.security.Krb5AndCertsSslSocketConnector$Krb5SslFilter.doFilter(Krb5AndCertsSslSocketConnector.java:221)
.......

It seems like the namenode is trying to get a kerberos ticket for the datanode (in hdfs-site.xml dfs.datanode.address is set to 0.0.0.0:1004<http://0.0.0.0:1004/> and dfs.datanode.http.address set to 0.0.0.0:1006<http://0.0.0.0:1006/>) but failed. I googled about 0.0.0.0, it is said to be something related to reverse DNS, from my node I can use 'host ip-address' to get the host name, so reverse DNS should be working. So what could've caused these errors?

Please give me some clue to this,
Thanks!
jack
Configuration:

added to core-site.xml:
<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value> <!-- A value of "simple" would disable security. -->
</property>

<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

to hdfs-site.xml:
  <!-- General HDFS security config -->
  <property>
    <name>dfs.block.access.token.enable</name>
    <value>true</value>
  </property>

  <!-- NameNode security config -->
  <property>
    <name>dfs.https.address</name>
    <value>c10i-bl0.us.oracle.com:50470<http://c10i-bl0.us.oracle.com:50470/></value>
  </property>
  <property>
    <name>dfs.https.port</name>
    <value>50470</value>
  </property>
  <property>
    <name>dfs.namenode.keytab.file</name>
    <value>/usr/hadoop/hadoop-1.0.1/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
  </property>
  <property>
    <name>dfs.namenode.kerberos.principal</name>
    <value>hdfs/[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]></value>
  </property>
  <property>
    <name>dfs.namenode.kerberos.https.principal</name>
    <value>host/[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]></value>
  </property>

  <!-- Secondary NameNode security config -->
  <property>
    <name>dfs.secondary.https.address</name>
    <value>c10i-bl0.us.oracle.com:50495<http://c10i-bl0.us.oracle.com:50495/></value>
  </property>
  <property>
    <name>dfs.secondary.https.port</name>
    <value>50495</value>
  </property>
  <property>
    <name>dfs.secondary.namenode.keytab.file</name>
    <value>/usr/hadoop/hadoop-1.0.1/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
  </property>
  <property>
    <name>dfs.secondary.namenode.kerberos.principal</name>
    <value>hdfs/[EMAIL PROTECTED]<mailto:HOST@CL