Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> sqoop import into secure Hbase with kerberos


+
Suhas Satish 2013-08-05, 19:15
+
Abraham Elmahrek 2013-08-05, 19:52
+
Suhas Satish 2013-08-05, 20:53
+
Abraham Elmahrek 2013-08-05, 21:29
+
Suhas Satish 2013-08-05, 22:55
+
Abraham Elmahrek 2013-08-05, 23:48
+
Suhas Satish 2013-08-06, 17:31
+
Suhas Satish 2013-08-06, 18:09
+
Abraham Elmahrek 2013-08-06, 18:13
+
Abraham Elmahrek 2013-08-06, 18:23
Copy link to this message
-
Re: sqoop import into secure Hbase with kerberos
Does this mean that sqoop tries to read  hbase-site.xml and then expectes
hbase to pass the  delegation token to it thru hbase.security.user class ?
I am using hbase 94.9
Hbase complains with the following msg -
2013-08-05 11:59:33,121 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.security.AccessDeniedException: Token generation
only allowed for Kerberos authenticated clients
at
org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)

What am I missing here? Should I specify anything in sqoop-site.xml
 related to kerberos?

Cheers,
Suhas.
On Tue, Aug 6, 2013 at 11:23 AM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:

> Sorry, apparently this is an HBase specific token. See here
> http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication.
>
>
> On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>
>> Suhas,
>>
>> Sqoop 1.4.3 simply fetches the authenticated user from credentials cache
>> and fetches a delegation token for HBase. See
>> https://issues.apache.org/jira/browse/SQOOP-599 for more information.
>>
>> -Abe
>>
>>
>> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>>
>>> I was able to isolate this problem to the Sqoop side not picking up
>>> correct kerberos credentials. Hbase is picking up the correct kerberos
>>> credentials when Hbase put and scan are done in isolation without using
>>> Sqoop.
>>>
>>> A direct map-reduce put into HBase uses the following 2 methods -
>>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
>>> TableMapReduceUtil.initCredentials(job);
>>>
>>> I was looking at how sqoop 1.4.3 does HBase puts to see if it converts
>>> sqoop import arguments into map-reduce jobs and uses the above methods
>>> somewhere. This is what I found -
>>> HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase
>>> "put" operation - has a method to get hadoop configuration, but none to
>>> merge any kerberos specific configurations specified  in sqoop-site.xml-
>>>
>>>   public Configuration getConf() {
>>>     return this.conf;
>>>
>>>
>>>
>>> HBaseUtil.java   - makes sure hbase jars are present on class path
>>> PutTransformer.java  - converts jdbc statements in the form of K-V map
>>> into hbase put commands and returns a list
>>> ToStringPutTransformer.java - extends the above class
>>>
>>>  Does anyone know sqoop internals of how to specify kerberos
>>> configurations and get sqoop to read them?
>>>
>>> Cheers,
>>> Suhas.
>>>
>>>
>>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>>>
>>>> Ataching the logs here at the time of authentication, I do not see any
>>>> error msges here.
>>>>
>>>> /var/log/kadmind.log
>>>> /var/log/krb5kdc.log
>>>>
>>>> Please let me know if there is any other places I can find other log
>>>> files
>>>>
>>>> Cheers,
>>>> Suhas.
>>>>
>>>>
>>>> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> User,
>>>>>
>>>>> Could you please provide your KDC logs around the time you tried to
>>>>> authenticate?
>>>>>
>>>>> Note: A kerberos client will negotiate the encryption algorithm it
>>>>> can/will use with the KDC. It may choose AES-256.
>>>>>
>>>>> -Abe
>>>>>
>>>>>
>>>>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> I generated a keytab with the following cmd and it supports multiple
>>>>>> encryption types other than aes256 as listed below.
>>>>>> But I still get the same error from sqoop import tool because the
>>>>>> sqoop.keytab is not being read (sqoop being the hbase client in this case).
>>>>>>
>>>>>> kadmin:  ktadd -k sqoop.keytab kuser1
>>>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>>>> aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>>>> aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>>>> Entry for principal kuser1 with kvno 2, encryption type des3-cbc-sha1
+
Jarek Jarcec Cecho 2013-08-11, 20:10
+
Suhas Satish 2013-08-11, 23:10
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB