Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Loading hbase-site.xml settings from Hadoop MR job


+
Dolan Antenucci 2013-09-23, 16:09
+
Shahab Yunus 2013-09-23, 16:53
+
Dolan Antenucci 2013-09-23, 17:40
+
Renato Marroquín Mogrovej... 2013-09-23, 17:06
+
Dolan Antenucci 2013-09-23, 17:42
+
Renato Marroquín Mogrovej... 2013-09-25, 00:08
Copy link to this message
-
Re: Loading hbase-site.xml settings from Hadoop MR job
Hi Renato,

I've double checked that the hbase-site.xml file exists on the client
server.

One oddity I noticed is that when I use HBaseAdmin(), the correct ZK server
is used (thus my hbase-site.xml file is being used); however, when I use
HTable() and HFileOutputFormat.configureIncrementalLoad(), the wrong ZK
server is used (thus hbase-site.xml is not being used).

// uses correct ZK server:
HBaseAdmin admin = new HBaseAdmin(conf);
admin.disableTable(tableName);
admin.deleteTable(tableName);

// uses WRONG ZK server:
HTable hTable = new HTable(conf, tableName);
HFileOutputFormat.configureIncrementalLoad(job, hTable);

These are in the same Hadoop job, run right after one another.  Could this
be a bug with HTable or HFileOutputFormat?

On Tue, Sep 24, 2013 at 8:08 PM, Renato Marroquín Mogrovejo <
[EMAIL PROTECTED]> wrote:

> Hi Dolan,
>
>
>
> 2013/9/23 Dolan Antenucci <[EMAIL PROTECTED]>
>
> > Hi Renato,
> >
> > Can you clarify your recommendation?
>
>
> Sorry about this. I will try to be more helpful (:
>
>
> > Currently I've added the directory
> > where my hbase-site.xml file lives (/etc/hbase/conf/) to my Hadoop
> > classpath (as described above).
>
>
> So this means in theory that all your Hadoop Cluster knows this
> hbase-site.xml, but this also means that the hbase-site.xml should be
> present in all nodes within your cluster.
>
>
> > Note: from the client machine (where I'm
> > starting my MR job), I generated the above class path by running "hadoop
> > classpath".
>
>
> Could you please double check that the hbase-site.xml is actually available
> for this client?
>
>
> > Also worth noting that the /etc/hbase/conf/hbase-site.xml file
> > on this client machine points to the correct ZK quorum.
> >
> > Thanks
> >
> >
> Hope I helped a little bit more.
>
>
> Renato M.
>
> >
> > On Mon, Sep 23, 2013 at 1:06 PM, Renato Marroquín Mogrovejo <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Maybe you should putting this configurations within your class path, so
> > it
> > > can be reached from your clients env.
> > >
> > >
> > > 2013/9/23 Shahab Yunus <[EMAIL PROTECTED]>
> > >
> > > > From where are you running your job? From which machine? This client
> > > > machine from where you are kicking of this job should have the
> > > > hbase-site.xml with the correct ZK info in it. It seems that your
> > > > client/job is having and issue picking up the right ZK, rather than
> the
> > > > services running on your non-local cluster.
> > > >
> > > > Regards,
> > > > Shahab
> > > >
> > > >
> > > > On Mon, Sep 23, 2013 at 12:09 PM, Dolan Antenucci <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > I'm having an issue where my Hadoop MR job for bulk loading data
> into
> > > > Hbase
> > > > > is not reading my hbase-site.xml file -- thus it tries to connect
> to
> > > > > Zookeeper on localhost.  This is on a cluster using CDH4 on Ubuntu
> > > 12.04.
> > > > >
> > > > > Here's the code where it attempts to connect to local zookeeper:
> > > > >     Configuration conf = new Configuration(); // from
> > > > > org.apache.hadoop.conf
> > > > >     Job job = new Job(conf);
> > > > >     HTable hTable = new HTable(conf, tableName);
> > > > >     HFileOutputFormat.configureIncrementalLoad(job, hTable);
> > > > >
> > > > > As suggested by another thread I came across, I've added
> > > > "/etc/hbase/conf/"
> > > > > to my HADOOP_CLASSPATH (in /etc/hadoop/conf/hadoop-env.sh),
> restarted
> > > > > services, but no improvement. Here is the full classpath:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> /usr/local/hadoop/lib/hadoop-lzo-0.4.17-SNAPSHOT.jar:/etc/hbase/conf/::/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/*:/usr/lib/hadoop-0.20-mapreduce/.//*
> > > > >
> > > > > Any thoughts on what the problem could be?