Hi,
I encountered Parse Error creating table in CLI:

[root@tyu-linux hive-0.5.0-bin]# bin/hive
Hive history file=/tmp/root/hive_job_log_root_201003240832_1752130264.txt
hive> SHOW TABLES;
OK
Time taken: 8.09 seconds
hive> CREATE EXTERNAL TABLE users(key string, state string, country string,
country_id int)
    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    >
    > WITH SERDEPROPERTIES (
    >
    > "hbase.key.type" = "string",
    >
    > "hbase.columns.mapping" = "info:state,info:country,info:country_id",
    >
    > "hbase.table.name" = "users"
    >
    > );
FAILED: Parse Error: line 2:0 cannot recognize input 'STORED' in table file
format specification

*Here is the tail of hive.log:*
2010-03-24 08:32:07,937 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2010-03-24 08:34:18,337 ERROR ql.Driver (SessionState.java:printError(248))
- FAILED: Parse Error: line 2:0 cannot recognize input 'STORED' in table
file format specification

org.apache.hadoop.hive.ql.parse.ParseException: line 2:0 cannot recognize
input 'STORED' in table file format specification

        at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:357)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:267)
        at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
        at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

If you know how to fix my query, please share.

Hi Ted,

It looks like your copy of Hive does not have the changes that implemented
support for integration with HBase. HBase support was committed to trunk on
March 12th, and currently it is only available on trunk. In order to use it
you need to  checkout (or update) the source from the svn repository.
Instructions describing how to do this are on the Hive wiki.

Carl

On Thu, Mar 25, 2010 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Hi,
> I encountered Parse Error creating table in CLI:
>
> [root@tyu-linux hive-0.5.0-bin]# bin/hive
> Hive history file=/tmp/root/hive_job_log_root_201003240832_1752130264.txt
> hive> SHOW TABLES;
> OK
> Time taken: 8.09 seconds
> hive> CREATE EXTERNAL TABLE users(key string, state string, country string,
> country_id int)
>     > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>     >
>     > WITH SERDEPROPERTIES (
>     >
>     > "hbase.key.type" = "string",
>     >
>     > "hbase.columns.mapping" = "info:state,info:country,info:country_id",
>     >
>     > "hbase.table.name" = "users"
>     >
>     > );
> FAILED: Parse Error: line 2:0 cannot recognize input 'STORED' in table file
> format specification
>
> *Here is the tail of hive.log:*
> 2010-03-24 08:32:07,937 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2010-03-24 08:34:18,337 ERROR ql.Driver (SessionState.java:printError(248))
> - FAILED: Parse Error: line 2:0 cannot recognize input 'STORED' in table
> file format specification
>
> org.apache.hadoop.hive.ql.parse.ParseException: line 2:0 cannot recognize
> input 'STORED' in table file format specification
>
>         at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:357)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:267)
>         at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> If you know how to fix my query, please share.
>

I got pass that error when I use CLI from trunk.

I copied hbase-site.xml from hbase master machine to trunk/dist/conf.

When I issue the following command in CLI:
CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat
int)
    STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    WITH SERDEPROPERTIES (
    "hbase.key.type" = "string",
    "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    "hbase.table.name" = "ruletable"
    );

I get:

2010-03-25 15:49:52,557 WARN  client.HConnectionManager$TableServers
(HConnectionManager.java:tableExists(411)) - Testing for table existence
threw exception
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 10.10.31.143:60020 for region .META.,,1, row '', but failed
after 10 attempts.
Exceptions:
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020 failed on local exception:
java.io.EOFException

        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
        at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
        at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.tableExists(HConnectionManager.java:404)
        at
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:113)
        at
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:100)
        at
org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:149)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:280)

However, I don't see any exception in hbase-hbaseadmin-regionserver.log on
10.10.31.143 within the past 4 hours.
No exception in hbase-hbaseadmin-master.log on hbase master machine either.

Please comment.

On Thu, Mar 25, 2010 at 11:22 AM, Carl Steinbach <[EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> It looks like your copy of Hive does not have the changes that implemented
> support for integration with HBase. HBase support was committed to trunk on
> March 12th, and currently it is only available on trunk. In order to use it
> you need to  checkout (or update) the source from the svn repository.
> Instructions describing how to do this are on the Hive wiki.
>
> Carl
>
>
> On Thu, Mar 25, 2010 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>> I encountered Parse Error creating table in CLI:
>>
>> [root@tyu-linux hive-0.5.0-bin]# bin/hive
>> Hive history file=/tmp/root/hive_job_log_root_201003240832_1752130264.txt
>> hive> SHOW TABLES;
>> OK
>> Time taken: 8.09 seconds
>> hive> CREATE EXTERNAL TABLE users(key string, state string, country
>> string,
>> country_id int)
>>     > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>>     >
>>     > WITH SERDEPROPERTIES (
>>     >
>>     > "hbase.key.type" = "string",
>>     >
>>     > "hbase.columns.mapping" = "info:state,info:country,info:country_id",

Don't copy hbase-site.xml; you don't want to set up a new HBase cluster.  Just set your hive conf to point to your existing HBase master:

hbase.master=hbase.yoyodyne.com:60000

http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage

JVS

On Mar 25, 2010, at 4:16 PM, Ted Yu wrote:

I got pass that error when I use CLI from trunk.

I copied hbase-site.xml from hbase master machine to trunk/dist/conf.

When I issue the following command in CLI:
CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat int)
    STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    WITH SERDEPROPERTIES (
    "hbase.key.type" = "string",
    "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    "hbase.table.name<http://hbase.table.name/>" = "ruletable"
    );

I get:

2010-03-25 15:49:52,557 WARN  client.HConnectionManager$TableServers (HConnectionManager.java:tableExists(411)) - Testing for table existence threw exception
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.10.31.143:60020<http://10.10.31.143:60020/> for region .META.,,1, row '', but failed after 10 attempts.
Exceptions:
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException

        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.tableExists(HConnectionManager.java:404)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:113)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:100)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:149)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:280)

However, I don't see any exception in hbase-hbaseadmin-regionserver.log on 10.10.31.143 within the past 4 hours.
No exception in hbase-hbaseadmin-master.log on hbase master machine either.

Please comment.

On Thu, Mar 25, 2010 at 11:22 AM, Carl Steinbach <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Ted,

It looks like your copy of Hive does not have the changes that implemented support for integration with HBase. HBase support was committed to trunk on March 12th, and currently it is only available on trunk. In order to use it you need to  checkout (or update) the source from the svn repository. Instructions describing how to do this are on the Hive wiki.

Carl
On Thu, Mar 25, 2010 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,
I encountered Parse Error creating table in CLI:

[root@tyu-linux hive-0.5.0-bin]# bin/hive
Hive history file=/tmp/root/hive_job_log_root_201003240832_1752130264.txt
hive> SHOW TABLES;
OK
Time taken: 8.09 seconds
hive> CREATE EXTERNAL TABLE users(key string, state string, country string,
country_id int)
    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    >
    > WITH SERDEPROPERTIES (
    >
    > "hbase.key.type" = "string",
    >
    > "hbase.columns.mapping" = "info:state,info:country,info:country_id",
    >
    > "hbase.table.name<http://hbase.table.name/>" = "users"
    >
    > );
FAILED: Parse Error: line 2:0 cannot recognize input 'STORED' in table file
format specification

Here is the tail of hive.log:
2010-03-24 08:32:07,937 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2010-03-24 08:34:18,337 ERROR ql.Driver (SessionState.java:printError(248))
- FAILED: Parse Error: line 2:0 cannot recognize input 'STORED' in table
file format specification

org.apache.hadoop.hive.ql.parse.ParseException: line 2:0 cannot recognize
input 'STORED' in table file format specification

        at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:357)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:267)
        at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
        at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl

Hmmm, what kind of HBase cluster are you running (standalone, pseudo-distributed, or distributed)?  If distributed, you may need to pass zookeeper connection info instead of hbase.master; I haven't tested out distributed yet but will be soon.

JVS

On Mar 25, 2010, at 4:43 PM, Ted Yu wrote:

I removed hbase-site.xml

[root@tyu-linux dist]# bin/hive -hiveconf hbase.master=snv-it-lin-006:60000
Hive history file=/tmp/root/hive_job_log_root_201003251614_1493474415.txt
hive> CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat int)
    >     STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    >     WITH SERDEPROPERTIES (
    >     "hbase.key.type" = "string",
    >     "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    >     "hbase.table.name<http://hbase.table.name/>" = "ruletable"
    >     );
FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
        at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:63)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:149)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:280)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:320)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:1445)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:124)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
)

But http://snv-it-lin-006:60010/master.jsp shows everything is normal.
And hbase.master.port is 60000 in hbase-default.xml

Some suggestions ?

Thanks

On Thu, Mar 25, 2010 at 4:21 PM, John Sichi <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Don't copy hbase-site.xml; you don't want to set up a new HBase cluster.  Just set your hive conf to point to your existing HBase master:

hbase.master=hbase.yoyodyne.com:60000<http://hbase.yoyodyne.com:60000/>

http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage

JVS

On Mar 25, 2010, at 4:16 PM, Ted Yu wrote:

I got pass that error when I use CLI from trunk.

I copied hbase-site.xml from hbase master machine to trunk/dist/conf.

When I issue the following command in CLI:
CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat int)
    STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    WITH SERDEPROPERTIES (
    "hbase.key.type" = "string",
    "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    "hbase.table.name<http://hbase.table.name/>" = "ruletable"
    );

I get:

2010-03-25 15:49:52,557 WARN  client.HConnectionManager$TableServers (HConnectionManager.java:tableExists(411)) - Testing for table existence threw exception
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.10.31.143:60020<http://10.10.31.143:60020/> for region .META.,,1, row '', but failed after 10 attempts.
Exceptions:
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException

        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.tableExists(HConnectionManager.java:404)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.

I'll update that page once I get our test cluster set up.

In the meantime, if you figure it out, please post to this mailing list.

Try hbase.zookeeper.quorum plus hbase.zookeeper.property.clientPort.

Hbase mailing list says those are the minimum required.

JVS

On Mar 25, 2010, at 7:12 PM, "Ted Yu" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

We use distributed mode because the cluster would be deployed to production.

I didn't find zookeeper commandline parameter on <http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage> http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage
Do I need to use an entry in hbase-site.xml ?

Our hbase cluster is based on 0.20.1
But I used hbase client 0.20.3 against it before and didn't see RetriesExhaustedException.

Thanks

On Thu, Mar 25, 2010 at 6:07 PM, John Sichi <<mailto:[EMAIL PROTECTED]>[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hmmm, what kind of HBase cluster are you running (standalone, pseudo-distributed, or distributed)?  If distributed, you may need to pass zookeeper connection info instead of hbase.master; I haven't tested out distributed yet but will be soon.

JVS

On Mar 25, 2010, at 4:43 PM, Ted Yu wrote:

I removed hbase-site.xml

[root@tyu-linux dist]# bin/hive -hiveconf hbase.master=snv-it-lin-006:60000
Hive history file=/tmp/root/hive_job_log_root_201003251614_1493474415.txt
hive> CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat int)
    >     STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    >     WITH SERDEPROPERTIES (
    >     "hbase.key.type" = "string",
    >     "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    >     "<http://hbase.table.name/>hbase.table.name<http://hbase.table.name>" = "ruletable"
    >     );
FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
        at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:63)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:149)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:280)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:320)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:1445)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:124)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
)

But <http://snv-it-lin-006:60010/master.jsp> http://snv-it-lin-006:60010/master.jsp shows everything is normal.
And hbase.master.port is 60000 in hbase-default.xml

Some suggestions ?

Thanks

On Thu, Mar 25, 2010 at 4:21 PM, John Sichi <<mailto:[EMAIL PROTECTED]>[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Don't copy hbase-site.xml; you don't want to set up a new HBase cluster.  Just set your hive conf to point to your existing HBase master:

hbase.master=hbase.yoyodyne.com:60000<http://hbase.yoyodyne.com:60000/>

<http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage>http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage

JVS

On Mar 25, 2010, at 4:16 PM, Ted Yu wrote:

I got pass that error when I use CLI from trunk.

I copied hbase-site.xml from hbase master machine to trunk/dist/conf.

When I issue the following command in CLI:
CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat int)
    STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    WITH SERDEPROPERTIES (
    "hbase.key.type" = "string",
    "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    "<http://hbase.table.name/>hbase.table.name<http://hbase.table.name>" = "ruletable"
    );

I get:

2010-03-25 15:49:52,557 WARN  client.HConnectionManager$TableServers (HConnectionManager.java:tableExists(411)) - Testing for table existence threw exception
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.10.31.143:60020<http://10.10.31.143:60020/> for region .META.,,1, row '', but failed after 10 attempts.
Exceptions:
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.IOException: Call to /10.10.31.143:60020<http://10.10.31.143:60020/> failed on local exception: java.io.EOFException
java.io.

Ted, when you reply, please reply-all to the list so that the conversation remains accessible to all.

Parameters are passed implicitly via the configuration object, not explicitly by name.

The unit test only specifies the master because that is all that is needed for a standalone server.

JVS
On Mar 25, 2010, at 9:18 PM, "Ted Yu" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

I searched source code tree under trunk and didn't see how hbase.master parameter is passed to HBaseStorageHandler.java

Also it looks like the unit test only specifies hostname for hbase.master:
"Z:\kindsight\hive\trunk\hbase-handler\src\test\org\apache\hadoop\hive\hbase\HBaseQTestUtil.java"(55,17):      conf.set("hbase.master", "local");

BTW I saw a lot of following exceptions:
2010-03-25 13:24:27,056 WARN  zookeeper.ClientCnxn (ClientCnxn.java:cleanup(1001)) - Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)

Thanks

On Thu, Mar 25, 2010 at 7:12 PM, Ted Yu <<mailto:[EMAIL PROTECTED]>[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
We use distributed mode because the cluster would be deployed to production.

I didn't find zookeeper commandline parameter on <http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage> http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage
Do I need to use an entry in hbase-site.xml ?

Our hbase cluster is based on 0.20.1
But I used hbase client 0.20.3 against it before and didn't see RetriesExhaustedException.

Thanks
On Thu, Mar 25, 2010 at 6:07 PM, John Sichi <<mailto:[EMAIL PROTECTED]>[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hmmm, what kind of HBase cluster are you running (standalone, pseudo-distributed, or distributed)?  If distributed, you may need to pass zookeeper connection info instead of hbase.master; I haven't tested out distributed yet but will be soon.

JVS

On Mar 25, 2010, at 4:43 PM, Ted Yu wrote:

I removed hbase-site.xml

[root@tyu-linux dist]# bin/hive -hiveconf hbase.master=snv-it-lin-006:60000
Hive history file=/tmp/root/hive_job_log_root_201003251614_1493474415.txt
hive> CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat int)
    >     STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    >     WITH SERDEPROPERTIES (
    >     "hbase.key.type" = "string",
    >     "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    >     "<http://hbase.table.name/>hbase.table.name<http://hbase.table.name>" = "ruletable"
    >     );
FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
        at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:63)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:149)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:280)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:320)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:1445)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:124)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
)

But <http://snv-it-lin-006:60010/master.jsp> http://snv-it-lin-006:60010/master.jsp shows everything is normal.
And hbase.master.port is 60000 in hbase-default.xml

Some suggestions ?

Thanks

On Thu, Mar 25, 2010 at 4:21 PM, John Sichi <<mailto:[EMAIL PROTECTED]>[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Don't copy hbase-site.xml; you don't want to set up a new HBase cluster.  Just set your hive conf to point to your existing HBase master:

hbase.master=hbase.yoyodyne.com:60000<http://hbase.yoyodyne.com:60000/>

<http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage>http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Usage

JVS

On Mar 25, 2010, at 4:16 PM, Ted Yu wrote:

I got pass that error when I use CLI from trunk.

I copied hbase-site.xml from hbase master machine to trunk/dist/conf.

When I issue the following command in CLI:
CREATE EXTERNAL TABLE ruletable(key string, exactmatch_cat string, lpm_cat int)
    STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    WITH SERDEPROPERTIES (
    "hbase.key.type" = "string",
    "hbase.columns.mapping" = "exactmatch_1.0:category,lpm_1.0:category",
    "<http://hbase.table.name/>hbase.table.name<http://hbase.table.name>" = "ruletable"
    );

I get:

2010-03-25 15:49:52,557 WARN  client.HConnectionManager$TableServers (HConnectionManage