Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - sqoop import into secure Hbase with kerberos


+
Suhas Satish 2013-08-05, 19:15
+
Abraham Elmahrek 2013-08-05, 19:52
+
Suhas Satish 2013-08-05, 20:53
+
Abraham Elmahrek 2013-08-05, 21:29
+
Suhas Satish 2013-08-05, 22:55
+
Abraham Elmahrek 2013-08-05, 23:48
+
Suhas Satish 2013-08-06, 17:31
Copy link to this message
-
Re: sqoop import into secure Hbase with kerberos
Suhas Satish 2013-08-06, 18:09
I was able to isolate this problem to the Sqoop side not picking up correct
kerberos credentials. Hbase is picking up the correct kerberos credentials
when Hbase put and scan are done in isolation without using Sqoop.

A direct map-reduce put into HBase uses the following 2 methods -
HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
TableMapReduceUtil.initCredentials(job);

I was looking at how sqoop 1.4.3 does HBase puts to see if it converts
sqoop import arguments into map-reduce jobs and uses the above methods
somewhere. This is what I found -
HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase "put"
operation - has a method to get hadoop configuration, but none to merge any
kerberos specific configurations specified  in sqoop-site.xml-

  public Configuration getConf() {
    return this.conf;

HBaseUtil.java   - makes sure hbase jars are present on class path
PutTransformer.java  - converts jdbc statements in the form of K-V map into
hbase put commands and returns a list
ToStringPutTransformer.java - extends the above class

Does anyone know sqoop internals of how to specify kerberos configurations
and get sqoop to read them?

Cheers,
Suhas.
On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:

> Ataching the logs here at the time of authentication, I do not see any
> error msges here.
>
> /var/log/kadmind.log
> /var/log/krb5kdc.log
>
> Please let me know if there is any other places I can find other log files
>
> Cheers,
> Suhas.
>
>
> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:
>
>> User,
>>
>> Could you please provide your KDC logs around the time you tried to
>> authenticate?
>>
>> Note: A kerberos client will negotiate the encryption algorithm it
>> can/will use with the KDC. It may choose AES-256.
>>
>> -Abe
>>
>>
>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>>
>>> I generated a keytab with the following cmd and it supports multiple
>>> encryption types other than aes256 as listed below.
>>> But I still get the same error from sqoop import tool because the
>>> sqoop.keytab is not being read (sqoop being the hbase client in this case).
>>>
>>> kadmin:  ktadd -k sqoop.keytab kuser1
>>> Entry for principal kuser1 with kvno 2, encryption type
>>> aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>> Entry for principal kuser1 with kvno 2, encryption type
>>> aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>> Entry for principal kuser1 with kvno 2, encryption type des3-cbc-sha1
>>> added to keytab WRFILE:sqoop.keytab.
>>> Entry for principal kuser1 with kvno 2, encryption type arcfour-hmac
>>> added to keytab WRFILE:sqoop.keytab.
>>> Entry for principal kuser1 with kvno 2, encryption type des-hmac-sha1
>>> added to keytab WRFILE:sqoop.keytab.
>>> Entry for principal kuser1 with kvno 2, encryption type des-cbc-md5
>>> added to keytab WRFILE:sqoop.keytab.
>>>
>>> Here are some more debug logs I obtained from kerberos -
>>>
>>> *kadmin:  getprinc kuser1*
>>> Principal: [EMAIL PROTECTED]
>>> Expiration date: [never]
>>> Last password change: Mon Aug 05 15:40:30 PDT 2013
>>> Password expiration date: [none]
>>> Maximum ticket life: 1 day 00:00:00
>>> Maximum renewable life: 0 days 00:00:00
>>> Last modified: Mon Aug 05 15:40:30 PDT 2013 (mapr/[EMAIL PROTECTED])
>>> Last successful authentication: [never]
>>> Last failed authentication: [never]
>>> Failed password attempts: 0
>>> Number of keys: 6
>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt
>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt
>>> Key: vno 2, des3-cbc-sha1, no salt
>>> Key: vno 2, arcfour-hmac, no salt
>>> Key: vno 2, des-hmac-sha1, no salt
>>> Key: vno 2, des-cbc-md5, no salt
>>> MKey: vno 1
>>> Attributes:
>>> Policy: [none]
>>>
>>> *getprinc hbase/qa-node133.qa.lab*
>>> Principal: hbase/[EMAIL PROTECTED]
>>> Expiration date: [never]
>>> Last password change: Mon Jul 29 19:17:46 PDT 2013
>>> Password expiration date: [none]
+
Abraham Elmahrek 2013-08-06, 18:13
+
Abraham Elmahrek 2013-08-06, 18:23
+
Suhas Satish 2013-08-06, 20:30
+
Jarek Jarcec Cecho 2013-08-11, 20:10
+
Suhas Satish 2013-08-11, 23:10