Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> sqoop import into secure Hbase with kerberos


Copy link to this message
-
Re: sqoop import into secure Hbase with kerberos
Sorry, apparently this is an HBase specific token. See here
http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication.
On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:

> Suhas,
>
> Sqoop 1.4.3 simply fetches the authenticated user from credentials cache
> and fetches a delegation token for HBase. See
> https://issues.apache.org/jira/browse/SQOOP-599 for more information.
>
> -Abe
>
>
> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>
>> I was able to isolate this problem to the Sqoop side not picking up
>> correct kerberos credentials. Hbase is picking up the correct kerberos
>> credentials when Hbase put and scan are done in isolation without using
>> Sqoop.
>>
>> A direct map-reduce put into HBase uses the following 2 methods -
>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
>> TableMapReduceUtil.initCredentials(job);
>>
>> I was looking at how sqoop 1.4.3 does HBase puts to see if it converts
>> sqoop import arguments into map-reduce jobs and uses the above methods
>> somewhere. This is what I found -
>> HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase
>> "put" operation - has a method to get hadoop configuration, but none to
>> merge any kerberos specific configurations specified  in sqoop-site.xml-
>>
>>   public Configuration getConf() {
>>     return this.conf;
>>
>>
>>
>> HBaseUtil.java   - makes sure hbase jars are present on class path
>> PutTransformer.java  - converts jdbc statements in the form of K-V map
>> into hbase put commands and returns a list
>> ToStringPutTransformer.java - extends the above class
>>
>>  Does anyone know sqoop internals of how to specify kerberos
>> configurations and get sqoop to read them?
>>
>> Cheers,
>> Suhas.
>>
>>
>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>>
>>> Ataching the logs here at the time of authentication, I do not see any
>>> error msges here.
>>>
>>> /var/log/kadmind.log
>>> /var/log/krb5kdc.log
>>>
>>> Please let me know if there is any other places I can find other log
>>> files
>>>
>>> Cheers,
>>> Suhas.
>>>
>>>
>>> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>>>
>>>> User,
>>>>
>>>> Could you please provide your KDC logs around the time you tried to
>>>> authenticate?
>>>>
>>>> Note: A kerberos client will negotiate the encryption algorithm it
>>>> can/will use with the KDC. It may choose AES-256.
>>>>
>>>> -Abe
>>>>
>>>>
>>>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> I generated a keytab with the following cmd and it supports multiple
>>>>> encryption types other than aes256 as listed below.
>>>>> But I still get the same error from sqoop import tool because the
>>>>> sqoop.keytab is not being read (sqoop being the hbase client in this case).
>>>>>
>>>>> kadmin:  ktadd -k sqoop.keytab kuser1
>>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>>> aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>>> aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type des3-cbc-sha1
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type arcfour-hmac
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type des-hmac-sha1
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type des-cbc-md5
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>>
>>>>> Here are some more debug logs I obtained from kerberos -
>>>>>
>>>>> *kadmin:  getprinc kuser1*
>>>>> Principal: [EMAIL PROTECTED]
>>>>> Expiration date: [never]
>>>>> Last password change: Mon Aug 05 15:40:30 PDT 2013
>>>>> Password expiration date: [none]
>>>>> Maximum ticket life: 1 day 00:00:00
>>>>> Maximum renewable life: 0 days 00:00:00