Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> sqoop import into secure Hbase with kerberos


+
Suhas Satish 2013-08-05, 19:15
+
Abraham Elmahrek 2013-08-05, 19:52
+
Suhas Satish 2013-08-05, 20:53
+
Abraham Elmahrek 2013-08-05, 21:29
+
Suhas Satish 2013-08-05, 22:55
+
Abraham Elmahrek 2013-08-05, 23:48
+
Suhas Satish 2013-08-06, 17:31
+
Suhas Satish 2013-08-06, 18:09
+
Abraham Elmahrek 2013-08-06, 18:13
+
Abraham Elmahrek 2013-08-06, 18:23
+
Suhas Satish 2013-08-06, 20:30
+
Jarek Jarcec Cecho 2013-08-11, 20:10
Copy link to this message
-
Re: sqoop import into secure Hbase with kerberos
I figured out that I had to authenticate using kinit. But hadoop UGI object
which does the authentication had  abug in it as  a result of which the
authentication wasn't happening. We are working on a fix in the hadoop
code.
Cheers,
Suhas.
On Sun, Aug 11, 2013 at 1:10 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Suhas,
> you should not be specifying anything in the sqoop-site.xml regarding
> kerberos. You should authenticate yourself (using kinit) and Sqoop will
> simply use those credentials to communicate with Hadoop and HBase.
>
> Would you mind sharing with us entire Sqoop command line and entire log
> generated with parameter --verbose?
>
> Jarcec
>
> On Tue, Aug 06, 2013 at 01:30:35PM -0700, Suhas Satish wrote:
> > Does this mean that sqoop tries to read  hbase-site.xml and then expectes
> > hbase to pass the  delegation token to it thru hbase.security.user class
> ?
> > I am using hbase 94.9
> > Hbase complains with the following msg -
> > 2013-08-05 11:59:33,121 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.security.AccessDeniedException: Token generation
> > only allowed for Kerberos authenticated clients
> > at
> >
> org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)
> >
> > What am I missing here? Should I specify anything in sqoop-site.xml
> >  related to kerberos?
> >
> > Cheers,
> > Suhas.
> >
> >
> > On Tue, Aug 6, 2013 at 11:23 AM, Abraham Elmahrek <[EMAIL PROTECTED]>
> wrote:
> >
> > > Sorry, apparently this is an HBase specific token. See here
> > > http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication.
> > >
> > >
> > > On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <[EMAIL PROTECTED]
> >wrote:
> > >
> > >> Suhas,
> > >>
> > >> Sqoop 1.4.3 simply fetches the authenticated user from credentials
> cache
> > >> and fetches a delegation token for HBase. See
> > >> https://issues.apache.org/jira/browse/SQOOP-599 for more information.
> > >>
> > >> -Abe
> > >>
> > >>
> > >> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[EMAIL PROTECTED]
> >wrote:
> > >>
> > >>> I was able to isolate this problem to the Sqoop side not picking up
> > >>> correct kerberos credentials. Hbase is picking up the correct
> kerberos
> > >>> credentials when Hbase put and scan are done in isolation without
> using
> > >>> Sqoop.
> > >>>
> > >>> A direct map-reduce put into HBase uses the following 2 methods -
> > >>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
> > >>> TableMapReduceUtil.initCredentials(job);
> > >>>
> > >>> I was looking at how sqoop 1.4.3 does HBase puts to see if it
> converts
> > >>> sqoop import arguments into map-reduce jobs and uses the above
> methods
> > >>> somewhere. This is what I found -
> > >>> HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase
> > >>> "put" operation - has a method to get hadoop configuration, but none
> to
> > >>> merge any kerberos specific configurations specified  in
> sqoop-site.xml-
> > >>>
> > >>>   public Configuration getConf() {
> > >>>     return this.conf;
> > >>>
> > >>>
> > >>>
> > >>> HBaseUtil.java   - makes sure hbase jars are present on class path
> > >>> PutTransformer.java  - converts jdbc statements in the form of K-V
> map
> > >>> into hbase put commands and returns a list
> > >>> ToStringPutTransformer.java - extends the above class
> > >>>
> > >>>  Does anyone know sqoop internals of how to specify kerberos
> > >>> configurations and get sqoop to read them?
> > >>>
> > >>> Cheers,
> > >>> Suhas.
> > >>>
> > >>>
> > >>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <
> [EMAIL PROTECTED]>wrote:
> > >>>
> > >>>> Ataching the logs here at the time of authentication, I do not see
> any
> > >>>> error msges here.
> > >>>>
> > >>>> /var/log/kadmind.log
> > >>>> /var/log/krb5kdc.log
> > >>>>
> > >>>> Please let me know if there is any other places I can find other log
> > >>>> files
> > >>>>
> > >>>> Cheers,
> > >>>> Suhas.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB