Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive Meta Server (Thrift Server) Failover / Redundancy / Load Balancing

Copy link to this message
Re: Hive Meta Server (Thrift Server) Failover / Redundancy / Load Balancing
Hi Manish,

Your understanding is mostly correct, though there is one additional bit.
MetastoreClient in current incarnation don't automatically reconnect in
case connection gets dropped for a connected session. As a result, it won't
failover active connections. New connections would be fine. Fortunately,
though there is a work in progress for this on :
https://issues.apache.org/jira/browse/HIVE-3400 In case you want to help
out you should help there.

For secure case, as you pointed out you additionally need ZooKeeper to
store security tokens. So, you need to bring up a ZK cluster. But, if you
think dedicating 3 nodes for ZK for metastore is an overhead then you would
need https://issues.apache.org/jira/browse/HIVE-3255 With that patch,
tokens are stored in same backend db, so there would be no need to bring up
ZK cluster.

Hopefully, both of these patches gets in for 0.10 release.


On Wed, Nov 7, 2012 at 11:28 PM, Manish Malhotra <

> Hi,
> I need to build a failover/LB solution for Hive Services.
> MySQL DB is fine, and can work out.
> But for Hive Metastore Service, can I simply put the Load Balancer like HA
> Proxy etc. in between the client and achieve this.
> Thrift Servers and default stateless, not sure about hive one.
> I red very few comments on this problem.
> Similar approach blogged at :
> http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/running_a_hive_thrift_cluster
> Very Imp from HCatalog mailing thread:
> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201109.mbox/%[EMAIL PROTECTED]%3E
> As per this mailing thread, if security is used in HIVE thrift meta server
> then need to do more modification in the server as it maintain the token of
> user for that session. (user connection).
> Please help me to move forward on this problem and please verify if my
> understanding is correct or not on the above 2 blogs / mail.
> Is there any initial work done under HCatalog or Hive, which I can look
> into and extend / patch.
> Regards,
> Manish