I worked with hadoop-1.2.1 which is fine working, but need with real requirement is Hadoop-2.2.0 i.e, YARN
Please show the way to get solve with apache products
I Installed HBase-0.96.1.1-hadoop2 on top of hadoop-2.2.0, installed hive-0.11.0 too,
Copied hive-hbase-handler-0.11.0 to hadoop-2.2.0, guva.jar, zookeeper.jar
I created hbase-0.96.1.1-hadoop2.jar(which is absent in version) and worked but not getting connected with hive. I suppose to work with hadoop-0.94.x but the version is not compatible with hadoop-2.2.0
I tried in all the way, not found the solution, please suggest me On Thu, Mar 13, 2014 at 4:20 PM, Joshi, Rekha <[EMAIL PROTECTED]>wrote:
Here is my document in an attached file. It contains my flow of hive-hbase integration. 1. I created a table in mysql and imported into hbase, 2. I copied a hive-handler.jar, guva.jar and zookeeper.jar file in Hadoop-2.2.0/share/mapred/lib folder 3. copied required hive(0.11.0) jars to hbase(0.96.1.1-hadoop2) && hbase jars to hive 4. Trying to create a table in hive with hbase properties but gets error as *FAILED: Error in metadata: MetaException(message:java.io.IOException: java.lang.reflect.InvocationTargetException* I think so hbase-0.96.1.1-hadoop2.jar file is not present in HBase directory.
IMPORT DATA WITH SQOOP in HDFS and HBASE , HIVE-HBASE INTEGRATION
To done this task, Tools that was used (Apache products) HADOOP-2.2.0
1. create a table in mysql
I selected a table with composite primary keys
CREATE TABLE widgets(id INT NOT NULL AUTO_INCREMENT, widget_name VARCHAR(64) NOT NULL, price DECIMAL(10,2), design_date DATE, version INT, design_comment VARCHAR(100), PRIMARY KEY(id,widget_name) ); 2. Insert data into table
INSERT INTO widgets VALUES (NULL, 'sprocket', 0.25, '2010-02-10', 1, 'Connects two gizmos'); INSERT INTO widgets VALUES (NULL, 'gizmo', 4.00, '2009-11-30', 4, NULL); INSERT INTO widgets VALUES (NULL, 'gadget', 99.99, '1983-08-13', 13, 'Our flagship product'); INSERT INTO widgets VALUES (NULL, 'database', 89.60, '1989-08-13', 13, 'entire data save here'); INSERT INTO widgets VALUES (NULL, 'BIGDATA', 100.00, '2014-01-20', 2, 'hadoop lead the world');
mysql> select * from widgets; +----+-------------+--------+-------------+---------+-----------------------+ | id | widget_name | price | design_date | version | design_comment | +----+-------------+--------+-------------+---------+-----------------------+ | 1 | sprocket | 0.25 | 2010-02-10 | 1 | Connects two gizmos | | 2 | gizmo | 4.00 | 2009-11-30 | 4 | NULL | | 3 | gadget | 99.99 | 1983-08-13 | 13 | Our flagship product | | 4 | database | 89.60 | 1989-10-13 | 13 | entire data save here | | 5 | BIGDATA | 100.00 | 2014-01-20 | 2 | hadoop lead the world | +----+-------------+--------+-------------+---------+-----------------------+ 5 rows in set (0.00 sec) exit; 3. Import data into hbase
create a table in hive that contains hbase configuration in-order to reflect data from hive to hbase or vice-versa.
CREATE external TABLE widgets(id int,widget_name string,price DOUBLE, design_date TIMESTAMP,version int, design_comment string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES("hbase.columns.mapping"=":key,cf:widget_name,cf:price,cf:design_date,cf:version,cf:design_comment")TBLPROPERTIES("hbase.table.name" = "widgets");
Add Jar files from hive and hbase HIVE_HOME/lib/hive-hbase-handler-VERSION.jar ----> ./HADOOP_HOME/share/mapreduce/lib HIVE_HOME/lib/hbase-core-VERSION.jar ----> ./HADOOP_HOME/share/mapreduce/lib
Also, added guva.jar, zookeeper.jar to mapred, hive and hbase In this process, I got error as
FAILED: Error in metadata: MetaException(message:java.io.IOException: java.lang.reflect.InvocationTargetException
This is because, hive-0.11.0/lib/ folder doesn't contain hbase-0.96.1.1-hadoop2.jar file.
In hive-0.11.0 or 0.12.0 consists of hbase-0.94.16.jar file. But We are using hbase-0.96.1.1-hadoop2 which is good compatable with YARN/hadoop-2.2.0 . So we need hbase-0.96.1.1-hadoop2.jar file in HIVE_HOME/lib inorder to overcome the error. Also I gone through readme files in hive/sqoop/hbase .
hbase-dev to bcc; adding hive-user. this is a question for the user lists, and more for Hive's than HBase, as HBaseStorageHandler is code in Hive project, not HBase.
You are embarking into a brave world. Because your aim is the interop across these different Apache projects, I highly recommend using a vendor's distribution. The reason being that the vendor has done the work of certifying interoperability for you, so you won't be left finding bugs in the edges that aren't well tested in the communities. Notice in the blog post referenced, I was using a distribution and even then I've found interop issues.
Hive 0.11 will require you manually specify the HBase jars. Please don't copy them around, instead use environment variables. You can follow along with my blog post and set HADOOP_CLASSPATH and HIVE_AUX_JARS_PATH appropriately. You're using HBase 0.96.1, which includes HBASE-8438, so you can use `hbase mapredcp` instead of itemizing jars manually. This requirement (hopefully) goes away entirely in Hive 0.13 with HIVE-2379 and HIVE-2055.
Finally, are you using Hive on the CLI or through a web tool (Hue?). The details change slightly based on all these... details.
Good luck. Keep asking questions. Please file bugs.
Thanks, Nick On Thu, Mar 13, 2014 at 10:56 PM, Sai Pavan Gadde <[EMAIL PROTECTED]>wrote:
We believe it is better than both. Its full-featured ANSI SQL compliant database with full transaction support based on mature database optimizer. On Mon, Mar 24, 2014 at 1:07 AM, Asaf Mesika <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext