Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # user - sqoop import into secure Hbase with kerberos


Copy link to this message
-
Re: sqoop import into secure Hbase with kerberos
Jarek Jarcec Cecho 2013-08-11, 20:10
Hi Suhas,
you should not be specifying anything in the sqoop-site.xml regarding kerberos. You should authenticate yourself (using kinit) and Sqoop will simply use those credentials to communicate with Hadoop and HBase.

Would you mind sharing with us entire Sqoop command line and entire log generated with parameter --verbose?

Jarcec

On Tue, Aug 06, 2013 at 01:30:35PM -0700, Suhas Satish wrote:
> Does this mean that sqoop tries to read  hbase-site.xml and then expectes
> hbase to pass the  delegation token to it thru hbase.security.user class ?
> I am using hbase 94.9
> Hbase complains with the following msg -
> 2013-08-05 11:59:33,121 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.security.AccessDeniedException: Token generation
> only allowed for Kerberos authenticated clients
> at
> org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)
>
> What am I missing here? Should I specify anything in sqoop-site.xml
>  related to kerberos?
>
> Cheers,
> Suhas.
>
>
> On Tue, Aug 6, 2013 at 11:23 AM, Abraham Elmahrek <[EMAIL PROTECTED]> wrote:
>
> > Sorry, apparently this is an HBase specific token. See here
> > http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication.
> >
> >
> > On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
> >
> >> Suhas,
> >>
> >> Sqoop 1.4.3 simply fetches the authenticated user from credentials cache
> >> and fetches a delegation token for HBase. See
> >> https://issues.apache.org/jira/browse/SQOOP-599 for more information.
> >>
> >> -Abe
> >>
> >>
> >> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:
> >>
> >>> I was able to isolate this problem to the Sqoop side not picking up
> >>> correct kerberos credentials. Hbase is picking up the correct kerberos
> >>> credentials when Hbase put and scan are done in isolation without using
> >>> Sqoop.
> >>>
> >>> A direct map-reduce put into HBase uses the following 2 methods -
> >>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
> >>> TableMapReduceUtil.initCredentials(job);
> >>>
> >>> I was looking at how sqoop 1.4.3 does HBase puts to see if it converts
> >>> sqoop import arguments into map-reduce jobs and uses the above methods
> >>> somewhere. This is what I found -
> >>> HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase
> >>> "put" operation - has a method to get hadoop configuration, but none to
> >>> merge any kerberos specific configurations specified  in sqoop-site.xml-
> >>>
> >>>   public Configuration getConf() {
> >>>     return this.conf;
> >>>
> >>>
> >>>
> >>> HBaseUtil.java   - makes sure hbase jars are present on class path
> >>> PutTransformer.java  - converts jdbc statements in the form of K-V map
> >>> into hbase put commands and returns a list
> >>> ToStringPutTransformer.java - extends the above class
> >>>
> >>>  Does anyone know sqoop internals of how to specify kerberos
> >>> configurations and get sqoop to read them?
> >>>
> >>> Cheers,
> >>> Suhas.
> >>>
> >>>
> >>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <[EMAIL PROTECTED]>wrote:
> >>>
> >>>> Ataching the logs here at the time of authentication, I do not see any
> >>>> error msges here.
> >>>>
> >>>> /var/log/kadmind.log
> >>>> /var/log/krb5kdc.log
> >>>>
> >>>> Please let me know if there is any other places I can find other log
> >>>> files
> >>>>
> >>>> Cheers,
> >>>> Suhas.
> >>>>
> >>>>
> >>>> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <[EMAIL PROTECTED]>wrote:
> >>>>
> >>>>> User,
> >>>>>
> >>>>> Could you please provide your KDC logs around the time you tried to
> >>>>> authenticate?
> >>>>>
> >>>>> Note: A kerberos client will negotiate the encryption algorithm it
> >>>>> can/will use with the KDC. It may choose AES-256.
> >>>>>
> >>>>> -Abe
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish <[EMAIL PROTECTED]>wrote:
> >>>>>
> >>>>>> I generated a keytab with the following cmd and it supports multiple