Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> sqoop import into secure Hbase with kerberos


Copy link to this message
-
Re: sqoop import into secure Hbase with kerberos
Suhas,

Sqoop 1.4.3 simply fetches the authenticated user from credentials cache
and fetches a delegation token for HBase. See
https://issues.apache.org/jira/browse/SQOOP-599 for more information.

-Abe
On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:

> I was able to isolate this problem to the Sqoop side not picking up
> correct kerberos credentials. Hbase is picking up the correct kerberos
> credentials when Hbase put and scan are done in isolation without using
> Sqoop.
>
> A direct map-reduce put into HBase uses the following 2 methods -
> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
> TableMapReduceUtil.initCredentials(job);
>
> I was looking at how sqoop 1.4.3 does HBase puts to see if it converts
> sqoop import arguments into map-reduce jobs and uses the above methods
> somewhere. This is what I found -
> HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase "put"
> operation - has a method to get hadoop configuration, but none to merge any
> kerberos specific configurations specified  in sqoop-site.xml-
>
>   public Configuration getConf() {
>     return this.conf;
>
>
>
> HBaseUtil.java   - makes sure hbase jars are present on class path
> PutTransformer.java  - converts jdbc statements in the form of K-V map
> into hbase put commands and returns a list
> ToStringPutTransformer.java - extends the above class
>
>  Does anyone know sqoop internals of how to specify kerberos
> configurations and get sqoop to read them?
>
> Cheers,
> Suhas.
>
>
> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>
>> Ataching the logs here at the time of authentication, I do not see any
>> error msges here.
>>
>> /var/log/kadmind.log
>> /var/log/krb5kdc.log
>>
>> Please let me know if there is any other places I can find other log
>> files
>>
>> Cheers,
>> Suhas.
>>
>>
>> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>>
>>> User,
>>>
>>> Could you please provide your KDC logs around the time you tried to
>>> authenticate?
>>>
>>> Note: A kerberos client will negotiate the encryption algorithm it
>>> can/will use with the KDC. It may choose AES-256.
>>>
>>> -Abe
>>>
>>>
>>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>>>
>>>> I generated a keytab with the following cmd and it supports multiple
>>>> encryption types other than aes256 as listed below.
>>>> But I still get the same error from sqoop import tool because the
>>>> sqoop.keytab is not being read (sqoop being the hbase client in this case).
>>>>
>>>> kadmin:  ktadd -k sqoop.keytab kuser1
>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>> aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>> aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>> Entry for principal kuser1 with kvno 2, encryption type des3-cbc-sha1
>>>> added to keytab WRFILE:sqoop.keytab.
>>>> Entry for principal kuser1 with kvno 2, encryption type arcfour-hmac
>>>> added to keytab WRFILE:sqoop.keytab.
>>>> Entry for principal kuser1 with kvno 2, encryption type des-hmac-sha1
>>>> added to keytab WRFILE:sqoop.keytab.
>>>> Entry for principal kuser1 with kvno 2, encryption type des-cbc-md5
>>>> added to keytab WRFILE:sqoop.keytab.
>>>>
>>>> Here are some more debug logs I obtained from kerberos -
>>>>
>>>> *kadmin:  getprinc kuser1*
>>>> Principal: [EMAIL PROTECTED]
>>>> Expiration date: [never]
>>>> Last password change: Mon Aug 05 15:40:30 PDT 2013
>>>> Password expiration date: [none]
>>>> Maximum ticket life: 1 day 00:00:00
>>>> Maximum renewable life: 0 days 00:00:00
>>>> Last modified: Mon Aug 05 15:40:30 PDT 2013 (mapr/[EMAIL PROTECTED])
>>>> Last successful authentication: [never]
>>>> Last failed authentication: [never]
>>>> Failed password attempts: 0
>>>> Number of keys: 6
>>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt
>>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt