First, I'd like to thank the community for the time and effort they put into sharing their knowledge...

A few weeks back I was able to configure a secure hadoop/hbase cluster (MIT 1.6.1 Kerberos on cluster) using a Windows Domain Controller/AD for the KDC.  I'm using hadoop 1.0.3 and hbase 0.92.1-security distributions.

Now I am trying setup my own Unix KDC (MIT 1.9.1 Kerberos) against that same cluster.  I know the cluster is configured correctly.  The only new piece to the puzzle is the Unix KDC.  The problem occurs when I start the namenode.  It is actually able to login my namenode principal into the KDC just fine.  I can see in the namenode main code that the HTTP Server as well as the RPC server has been created successfully.  It's in the startTrashEmptier() method where the error occurs.  It's like Hadoop is acting as a client and connecting back into itself (hdfs service) when it receives a checksum error:

12/06/29 15:56:13 INFO security.UserGroupInformation: Login successful for user host/[EMAIL PROTECTED] using keytab file /etc/krb5.keytab
12/06/29 15:56:13 INFO ipc.Server: IPC Server Responder: starting
12/06/29 15:56:13 INFO ipc.Server: IPC Server listener on 8020: starting
Found key for host/[EMAIL PROTECTED](18)
Found key for host/[EMAIL PROTECTED](3)
Found key for host/[EMAIL PROTECTED](16)
Found key for host/[EMAIL PROTECTED](17)
Found key for host/[EMAIL PROTECTED](23)
Entered Krb5Context.acceptSecContext with state=STATE_NEW
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
Checksum failed !
12/06/29 15:56:13 INFO ipc.Server: IPC Server listener on 8020: readAndProcess threw exception javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]. Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
        at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
        at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1007)
        at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1180)
        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:537)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:344)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
        at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
        ... 7 more
Caused by: KrbException: Checksum failed
        at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:85)
        at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:77)
        at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:168)
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:268)
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
        ... 10 more
Caused by: java.security.GeneralSecurityException: Checksum failed
        at sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:388)
        at sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:74)
        at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:83)
        ... 16 more

I think it has something to do with the keys in my keytab.  Although, I can kinit into the KDC with all of the principals in my keytab so I don't know what the problem is.

I read something (not validated though) that there may be some incompatibility with Hadoop security and MIT 1.9.1.

Any insight here would be greatly appreciated.

Thanks.
Tony Dean
SAS Institute Inc.
Senior Software Developer
919-531-6704

On Fri, Jun 29, 2012 at 1:50 PM, Tony Dean <[EMAIL PROTECTED]> wrote:

>  First, I’d like to thank the community for the time and effort they put
> into sharing their knowledge…
>

Which version of Hadoop are you running? Which JDK are you using? You
probably need HDFS-2617 and JDK 1.6.0_31.

-- Owen

Hadoop 1.0.3, JDK1.6.0_21 with JCE export jars for strong encryption.

-----Original Message-----
From: Owen O'Malley [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 29, 2012 5:02 PM
To: [EMAIL PROTECTED]
Subject: Re: hadoop kerberos security / unix kdc

On Fri, Jun 29, 2012 at 1:50 PM, Tony Dean <[EMAIL PROTECTED]> wrote:

>  First, I’d like to thank the community for the time and effort they
> put into sharing their knowledge…
>

Which version of Hadoop are you running? Which JDK are you using? You probably need HDFS-2617 and JDK 1.6.0_31.

-- Owen

On Fri, Jun 29, 2012 at 2:07 PM, Tony Dean <[EMAIL PROTECTED]> wrote:

> Hadoop 1.0.3, JDK1.6.0_21 with JCE export jars for strong encryption.
You need to move up to a JDK > 1.6.0_27. I'd suggest 1.6.0_31.

For details, look at: http://wiki.apache.org/hadoop/HadoopJavaVersions

-- Owen

I installed 1.6.0 update 33 ... it didn't help this situation.

-----Original Message-----
From: Owen O'Malley [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 29, 2012 5:28 PM
To: [EMAIL PROTECTED]
Subject: Re: hadoop kerberos security / unix kdc

On Fri, Jun 29, 2012 at 2:07 PM, Tony Dean <[EMAIL PROTECTED]> wrote:

> Hadoop 1.0.3, JDK1.6.0_21 with JCE export jars for strong encryption.
You need to move up to a JDK > 1.6.0_27. I'd suggest 1.6.0_31.

For details, look at: http://wiki.apache.org/hadoop/HadoopJavaVersions

-- Owen

I have been looking at this for 2 days now with no avail... does anyone know why I would be getting a checksum error when I have validated my keys.

I actually deleted my service principals from kdc DB and added them back with a human readable password instead of random key.  I regenerated my keytab with those service principal.  From namenode, I am able to kinit to the kdc with and without the keytab.  However, when I start the namenode, I still get checksum.  I even tried a different kdc (older 1.8 instead of new 1.9.1) and received the same exception.

It has to be something simple, but I just can't figure it out.

If anyone has any ideas please let me know.

The latest traces are as follows:

Found key for host/[EMAIL PROTECTED](23)
Found key for host/[EMAIL PROTECTED](18)
Found ticket for host/[EMAIL PROTECTED] to go to krbtgt/[EMAIL PROTECTED] expiring on Mon Jul 02 00:33:02 EDT 2012
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for host/[EMAIL PROTECTED] to go to krbtgt/[EMAIL PROTECTED] expiring on Mon Jul 02 00:33:02 EDT 2012
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 3 1 23 16 17 18.
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
Checksum failed !
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
>>> KrbKdcReq send: kdc=cikdc.unx.sas.com UDP:88, timeout=30000, number of retries =3, #bytes=716
>>> KDCCommunication: kdc=cikdc.unx.sas.com UDP:88, timeout=30000,Attempt =1, #bytes=716
12/07/01 00:33:05 INFO ipc.Server: IPC Server listener on 8020: readAndProcess threw exception javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]. Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
        at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
        at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1007)
        at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1180)
        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:537)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:344)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
        at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
        ... 7 more
Caused by: KrbException: Checksum failed
        at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:85)
        at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:77)
        at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:168)
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:268)
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
        ... 10 more
Caused by: java.security.GeneralSecurityException: Checksum failed
        at sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:388)
        at sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:74)
        at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:83)
        ... 16 more

Thanks!

_____________________________________________
From: Tony Dean
Sent: Friday, June 29, 2012 4:50 PM
To: '[EMAIL PROTECTED]'
Subject: hadoop kerberos security / unix kdc
First, I'd like to thank the community for the time and effort they put into sharing their knowledge...

A few weeks back I was able to configure a secure hadoop/hbase cluster (MIT 1.6.1 Kerberos on cluster) using a Windows Domain Controller/AD for the KDC.  I'm using hadoop 1.0.3 and hbase 0.92.1-security distributions.

Now I am trying setup my own Unix KDC (MIT 1.9.1 Kerberos) against that same cluster.  I know the cluster is configured correctly.  The only new piece to the puzzle is the Unix KDC.  The problem occurs when I start the namenode.  It is actually able to login my namenode principal into the KDC just fine.  I can see in the namenode main code that the HTTP Server as well as the RPC server has been created successfully.  It's in the startTrashEmptier() method where the error occurs.  It's like Hadoop is acting as a client and connecting back into itself (hdfs service) when it receives a checksum error:

12/06/29 15:56:13 INFO security.UserGroupInformation: Login successful for user host/[EMAIL PROTECTED] using keytab file /etc/krb5.keytab
12/06/29 15:56:13 INFO ipc.Server: IPC Server Responder: starting
12/06/29 15:56:13 INFO ipc.Server: IPC Server listener on 8020: starting
Found key for host/[EMAIL PROTECTED](18)
Found key for host/[EMAIL PROTECTED](3)
Found key for host/rdcesx10030.race.sas.com@OBS