Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - sqoop import into secure Hbase with kerberos


+
Suhas Satish 2013-08-05, 19:15
+
Abraham Elmahrek 2013-08-05, 19:52
+
Suhas Satish 2013-08-05, 20:53
+
Abraham Elmahrek 2013-08-05, 21:29
+
Suhas Satish 2013-08-05, 22:55
+
Abraham Elmahrek 2013-08-05, 23:48
+
Suhas Satish 2013-08-06, 17:31
+
Suhas Satish 2013-08-06, 18:09
+
Abraham Elmahrek 2013-08-06, 18:13
+
Abraham Elmahrek 2013-08-06, 18:23
+
Suhas Satish 2013-08-06, 20:30
+
Jarek Jarcec Cecho 2013-08-11, 20:10
Copy link to this message
-
Re: sqoop import into secure Hbase with kerberos
Suhas Satish 2013-08-11, 23:10
I figured out that I had to authenticate using kinit. But hadoop UGI object
which does the authentication had  abug in it as  a result of which the
authentication wasn't happening. We are working on a fix in the hadoop
code.
Cheers,
Suhas.
On Sun, Aug 11, 2013 at 1:10 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Suhas,
> you should not be specifying anything in the sqoop-site.xml regarding
> kerberos. You should authenticate yourself (using kinit) and Sqoop will
> simply use those credentials to communicate with Hadoop and HBase.
>
> Would you mind sharing with us entire Sqoop command line and entire log
> generated with parameter --verbose?
>
> Jarcec
>
> On Tue, Aug 06, 2013 at 01:30:35PM -0700, Suhas Satish wrote:
> > Does this mean that sqoop tries to read  hbase-site.xml and then expectes
> > hbase to pass the  delegation token to it thru hbase.security.user class
> ?
> > I am using hbase 94.9
> > Hbase complains with the following msg -
> > 2013-08-05 11:59:33,121 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.security.AccessDeniedException: Token generation
> > only allowed for Kerberos authenticated clients
> > at
> >
> org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)
> >
> > What am I missing here? Should I specify anything in sqoop-site.xml
> >  related to kerberos?
> >
> > Cheers,
> > Suhas.
> >
> >
> > On Tue, Aug 6, 2013 at 11:23 AM, Abraham Elmahrek <[EMAIL PROTECTED]>
> wrote:
> >
> > > Sorry, apparently this is an HBase specific token. See here
> > > http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication.
> > >
> > >
> > > On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <[EMAIL PROTECTED]
> >wrote:
> > >
> > >> Suhas,
> > >>
> > >> Sqoop 1.4.3 simply fetches the authenticated user from credentials
> cache
> > >> and fetches a delegation token for HBase. See
> > >> https://issues.apache.org/jira/browse/SQOOP-599 for more information.
> > >>
> > >> -Abe
> > >>
> > >>
> > >> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[EMAIL PROTECTED]
> >wrote:
> > >>
> > >>> I was able to isolate this problem to the Sqoop side not picking up
> > >>> correct kerberos credentials. Hbase is picking up the correct
> kerberos
> > >>> credentials when Hbase put and scan are done in isolation without
> using
> > >>> Sqoop.
> > >>>
> > >>> A direct map-reduce put into HBase uses the following 2 methods -
> > >>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
> > >>> TableMapReduceUtil.initCredentials(job);
> > >>>
> > >>> I was looking at how sqoop 1.4.3 does HBase puts to see if it
> converts
> > >>> sqoop import arguments into map-reduce jobs and uses the above
> methods
> > >>> somewhere. This is what I found -
> > >>> HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase
> > >>> "put" operation - has a method to get hadoop configuration, but none
> to
> > >>> merge any kerberos specific configurations specified  in
> sqoop-site.xml-
> > >>>
> > >>>   public Configuration getConf() {
> > >>>     return this.conf;
> > >>>
> > >>>
> > >>>
> > >>> HBaseUtil.java   - makes sure hbase jars are present on class path
> > >>> PutTransformer.java  - converts jdbc statements in the form of K-V
> map
> > >>> into hbase put commands and returns a list
> > >>> ToStringPutTransformer.java - extends the above class
> > >>>
> > >>>  Does anyone know sqoop internals of how to specify kerberos
> > >>> configurations and get sqoop to read them?
> > >>>
> > >>> Cheers,
> > >>> Suhas.
> > >>>
> > >>>
> > >>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <
> [EMAIL PROTECTED]>wrote:
> > >>>
> > >>>> Ataching the logs here at the time of authentication, I do not see
> any
> > >>>> error msges here.
> > >>>>
> > >>>> /var/log/kadmind.log
> > >>>> /var/log/krb5kdc.log
> > >>>>
> > >>>> Please let me know if there is any other places I can find other log
> > >>>> files
> > >>>>
> > >>>> Cheers,
> > >>>> Suhas.