Adding hive-user.

On Thu, Mar 25, 2010 at 9:38 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> If I use the following commandline, I would see RetriesExhaustedException
> as I reported earlier this afternoon:
> [root@tyu-linux dist]# bin/hive -hiveconf hbase.master=snv-it-lin-006
> -hiveconf
> hbase.zookeeper.quorum=snv-it-lin-010,snv-it-lin-011,snv-it-lin-012
> -hiveconf hbase.zookeeper.property.clientPort=2181
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
> region server 10.10.31.136:60020 for region .META.,,1, row '', but failed
> after 10 attempts.
>
> On Thu, Mar 25, 2010 at 9:18 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> I searched source code tree under trunk and didn't see how hbase.master
>> parameter is passed to HBaseStorageHandler.java
>>
>> Also it looks like the unit test only specifies hostname for hbase.master:
>> "Z:\kindsight\hive\trunk\hbase-handler\src\test\org\apache\hadoop\hive\hbase\HBaseQTestUtil.java"(55,17):
>> conf.set("hbase.master", "local");
>>
>> BTW I saw a lot of following exceptions:
>> 2010-03-25 13:24:27,056 WARN  zookeeper.ClientCnxn
>> (ClientCnxn.java:cleanup(1001)) - Ignoring exception during shutdown input
>> java.nio.channels.ClosedChannelException
>>         at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>         at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>         at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>         at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>
>> Thanks
>>
>>
>> On Thu, Mar 25, 2010 at 7:12 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>>> We use distributed mode because the cluster would be deployed to
>>> production.
>>>
>>> I didn't find zookeeper commandline parameter on
>>> http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage
>>> Do I need to use an entry in hbase-site.xml ?
>>>
>>> Our hbase cluster is based on 0.20.1
>>> But I used hbase client 0.20.3 against it before and didn't see
>>> RetriesExhaustedException.
>>>
>>> Thanks
>>>
>>>
>>> On Thu, Mar 25, 2010 at 6:07 PM, John Sichi <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hmmm, what kind of HBase cluster are you running (standalone,
>>>> pseudo-distributed, or distributed)?  If distributed, you may need to pass
>>>> zookeeper connection info instead of hbase.master; I haven't tested out
>>>> distributed yet but will be soon.
>>>>
>>>> JVS
>>>>
>>>> On Mar 25, 2010, at 4:43 PM, Ted Yu wrote:
>>>>
>>>> I removed hbase-site.xml
>>>>
>>>> [root@tyu-linux dist]# bin/hive -hiveconf
>>>> hbase.master=snv-it-lin-006:60000
>>>> Hive history
>>>> file=/tmp/root/hive_job_log_root_201003251614_1493474415.txt
>>>> hive> CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string,
>>>> lpm_cat int)
>>>>     >     STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>>>>     >     WITH SERDEPROPERTIES (
>>>>     >     "hbase.key.type" = "string",
>>>>     >     "hbase.columns.mapping" >>>> "exactmatch_1.0:category,lpm_1.0:category",
>>>>     >     "hbase.table.name" = "ruletable"
>>>>     >     );
>>>> FAILED: Error in metadata:
>>>> MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException
>>>>         at
>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
>>>>         at
>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
>>>>         at
>>>> org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:63)
>>>>         at
>>>> org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:149)
>>>>         at
>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:280)
>>>>         at
>>>> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:320)
>>>>         at
>>>> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:1445)
>>>>         at
>>>> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:124)

Hi Ted,

I got our HBase test cluster set up and encountered the same problems you did at first, but I've got it working now.  Here's what I found out (I have updated the wiki accordingly).

1.  You MUST use the same HBase version consistently for client and server.  Most likely this is your problem.  I was initially trying with an 0.20.4-dev version on the cluster and 0.20.3 on the Hive client, and consistently getting MasterNotRunningException.  Then I rebuilt the handler against 0.20.4-dev and ran with that, and problem solved.  So delete the jars from hbase-handler/lib, replace them with the ones which correspond to your HBase version, and then run ant clean and ant package again.  (If you get build failures, you'll have to upgrade your HBase cluster or figure out how to make the handler backward compatible.)  Browsing the HBase JIRA, I found that the RPC protocol has changed quite a bit, so I guess this is the reason for the incompatibilities (it would be nice if the handshake failed fast with an accurate error message instead of saying that the master is not running when in fact it is).

2.  The only configuration necessary is to set hbase.zookeeper.quorum.  Do not set hbase.master for talking to a cluster.  And you don't need to set hbase.zookeeper.property.clientPort (assuming you are using the default on the zookeeper servers).

3.  Make sure there is no firewall weirdness in between client and server.  I hit this when I tested from my laptop over VPN to the cluster.  In the firewall case, MasterNotRunningException had null for its message, whereas in the inconsistent client/server version case, MasterNotRunningException had the correct IP:port for its message.

Let me know how it goes.

JVS