Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Re: How can I access secure HBase in UDF


+
Rohini Palaniswamy 2012-09-27, 17:54
Copy link to this message
-
Re: How can I access secure HBase in UDF
If someone figures this out ll the way to working code, could you blog it?
:)

D

On Thu, Sep 27, 2012 at 10:54 AM, Rohini Palaniswamy <
[EMAIL PROTECTED]> wrote:

> Ray,
>    In the frontend, you can do a new JobConf(HBaseConfiguration.create())
> and pass that to TableMapReduceUtil.initCredentials(). It will fetch the
> credentials and add to JobConf.credentials. You can then loop through
> jobConf.getCredentials().getAllTokens() and get the token of kind
> HBASE_AUTH_TOKEN and add it to UDFContext. On the backend you need to get
> the token from UDFContext and add it to the JobConf you create HTable with.
>
> Regards,
> Rohini
>
> On Thu, Sep 27, 2012 at 4:19 AM, Ray <[EMAIL PROTECTED]> wrote:
>
> > Thank you guys for your replies.
> >
> > TableMapReduceUtil.initCredentials() needs a Job object as argument not a
> > Configuration object returned by UDFContext.getUDFContext().getJobConf.
> >
> > I didn't dive into the hbase credentials very deeply, so I don't know how
> > to create a credential file and cache it.
> > Only I know is to use some utilities provided by hadoop and hbase to
> create
> > credentials for Job like TableMapReduceUtil.initCredentials().
> >
> > But to use the utilities, I need get the Job object when it's being
> > constructed in backend like the setLocation() of LoadFunc. Then the
> > credentials can take effect when job run.
> > I guess I can't do this in EvalFunc.
> >
> >
> > On Wed, Sep 26, 2012 at 11:26 PM, Ashutosh Chauhan <[EMAIL PROTECTED]
> > >wrote:
> >
> > > I will suggest to use HBase api to manage your credentials, instead
> > > handling them of your own.
> > >
> > >
> >
> TableMapReduceUtil.initCredentials(UDFContext.getUDFContext().getJobConf);
> > >
> > > Ashutosh
> > > On Tue, Sep 25, 2012 at 10:38 PM, Rohini Palaniswamy <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > That should work too. Just a note that UDFContext is serialized in
> > > JobConf
> > > > and would be visible to all (though base64 encoded) through
> Jobtracker
> > UI
> > > > unless acls for viewing are set. If you are not that paranoid about
> > > > security as in Yahoo! then it should be fine.
> > > >
> > > > -Rohini
> > > >
> > > > On Tue, Sep 25, 2012 at 5:09 PM, Alan Gates <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > You can use the UDFContext to pass information for the UDF in the
> > > JobConf
> > > > > without writing files.
> > > > >
> > > > > Alan.
> > > > >
> > > > > On Sep 25, 2012, at 10:48 AM, Rohini Palaniswamy wrote:
> > > > >
> > > > > > Ray,
> > > > > >   Looking at the EvalFunc interface, I can not see a way or
> > loophole
> > > to
> > > > > do
> > > > > > it.  EvalFunc does not have a reference to Job or JobConf object
> to
> > > add
> > > > > > credentials to it. It has getCacheFiles() to add files to
> > > > > DistributedCache,
> > > > > > but no method to add credentials. We should probably add one. The
> > not
> > > > so
> > > > > > nice workaround I can think of is to add the token as a file
> > > > > > to DistributedCache using getCacheFiles() and read the file
> > yourself
> > > in
> > > > > > EvalFunc and use it in communication with HBase.
> > > > > >
> > > > > > Regards,
> > > > > > Rohini
> > > > > >
> > > > > > On Tue, Sep 25, 2012 at 1:51 AM, Ray <[EMAIL PROTECTED]> wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> I have a requirement to access HBase in UDF. But the HBase is
> > > > > configured to
> > > > > >> be secure, which needs a credential when being connected in a
> > > > mapreduce
> > > > > >> job.
> > > > > >> I see you have added support of secure HBase in HBaseStorage
> > > > > >>
> > > > >
> > >
> https://issues.apache.org/jira/browse/PIG-2821?attachmentSortBy=dateTime
> > > > > >> But my UDF is an EvalFunc not Loader.
> > > > > >>
> > > > > >> Could you tell me how I can achieve this? Or is there any way I
> > can
> > > > add
> > > > > the
> > > > > >> credential in Job when the job is setup in backend?
> > > > > >>
> > > > > >> Thanks,
+
Ray 2012-10-08, 09:28
+
Rohini Palaniswamy 2012-11-05, 21:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB