Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Dual Hadoop/HBase configuration through same client


Copy link to this message
-
Re: Dual Hadoop/HBase configuration through same client
Damien Hardy 2013-04-27, 06:51
Hello

Maybe you should look at export tools source code as it can export HBase
data to distant HDFS space (setting a full hdfs:// url in command line
option for outputdir)

https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Export.java

Cheers,
2013/4/27 Shahab Yunus <[EMAIL PROTECTED]>

> Thanks Ted for the response. But the issue is that I want to read from one
> cluster and write to another. If I will have to clients then how will they
> communicate with each other? Essentially what am I trying to do here is
> intra-cluster data copy/exchange. Any other ideas or suggestions? Even if
> both servers have no security or one has Kerberos or both have
> authentication how to exchange data between them?
>
> I was actually not expecting that I cannot load multiple Hadoop or HBase
> configurations in 2 different Configuration objects in one application.
> As mentioned I have tried overwriting properties as well but
> security/authentication properties are overwritten somehow.
>
> Regards,
> Shahab
>
>
> On Fri, Apr 26, 2013 at 7:43 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Looks like the easiest solution is to use separate clients, one for each
> > cluster you want to connect to.
> >
> > Cheers
> >
> > On Sat, Apr 27, 2013 at 6:51 AM, Shahab Yunus <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hello,
> > >
> > > This is a follow-up to my previous post a few days back. I am trying to
> > > connect to 2 different Hadoop clusters' setups through a same client
> but
> > I
> > > am running into the issue that the config of one overwrites the other.
> > >
> > > The scenario is that I want to read data from an HBase table from one
> > > cluster and write it as a file on HDFS on the other. Individually, if I
> > try
> > > to write to them they both work but when I try this through a same Java
> > > client, they fail.
> > >
> > > I have tried loading the core-site.xml through addResource method of
> the
> > > Configuration class but only the first found config file is picked? I
> > have
> > > also tried by renaming the config files and then adding them as a
> > resource
> > > (again through the addResource method).
> > >
> > > The situation is compounded by the fact that one cluster is using
> > Kerberos
> > > authentication and the other is not? If the Kerberos server's file is
> > found
> > > first then authentication failures are faced for the other server when
> > > Hadoop tries to find client authentication information. If the 'simple'
> > > cluster's config is loaded first then 'Authentication is Required'
> error
> > is
> > > encountered against the Kerberos server.
> > >
> > > I will gladly provide more information. Is it even possible even if let
> > us
> > > say both servers have same security configuration or none? Any ideas?
> > > Thanks a million.
> > >
> > > Regards,
> > > Shahab
> > >
> >
>

--
Damien HARDY
IT Infrastructure Architect
Viadeo - 30 rue de la Victoire - 75009 Paris - France