Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> How to load hive metadata from conf dir


+
Parag Sarda 2013-02-11, 22:59
+
Mark Grover 2013-02-12, 04:57
+
Parag Sarda 2013-02-12, 08:18
+
Nitin Pawar 2013-02-12, 08:25
+
Parag Sarda 2013-02-12, 16:36
+
Dean Wampler 2013-02-12, 14:41
+
Edward Capriolo 2013-02-12, 16:53
Copy link to this message
-
Re: How to load hive metadata from conf dir
Hive-thrift is definitely best option till now. That said, I am wondering
if its possible to load megastore in local mode[1] to avoid dependency on
external service. Can I read the HIVE_CONF_DIR for javax.jdo.option.*
parameters and talk to sql server hosting hive metadata?

[1] https://cwiki.apache.org/Hive/adminmanual-metastoreadmin.html

Thanks,
Parag
On 12/02/13 10:23 PM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:

>If you use hive-thrift/hive-service you can get the location of a
>table through the Table API (instead of Dean's horrid bash-isms)
>
>http://hive.apache.org/docs/r0.7.0/api/org/apache/hadoop/hive/metastore/ap
>i/ThriftHiveMetastore.Client.html#get_table(java.lang.String,
>java.lang.String)
>
>Table t = ....
>t.getSd().getLocation()
>
>
>On Tue, Feb 12, 2013 at 9:41 AM, Dean Wampler
><[EMAIL PROTECTED]> wrote:
>> I'll mention another bash hack that I use all the time:
>>
>> hive -e 'some_command' | grep for_what_i_want |
>> sed_command_to_remove_just_i_dont_want
>>
>> For example, the following command will print just the value of
>> hive.metastore.warehouse.dir, sending all the logging junk written to
>>stderr
>> to /dev/null and stripping off the leading
>>"hive.metastore.warehouse.dir="
>> from the stdout output:
>>
>> hive -e 'set hive.metastore.warehouse.dir;' 2> /dev/null | sed -e
>> 's/hive.metastore.warehouse.dir=//'
>>
>> (No grep subcommand required in this case...)
>>
>> You could do something similar with DESCRIBE EXTENDED table PARTION(...)
>> Suppose you want a script that works for any property. Put the
>>following in
>> a script file, say hive-prop.sh:
>>
>> #!/bin/sh
>> hive -e "set $1;" 2> /dev/null | sed -e "s/$1=//"
>>
>> Make it executable (chmod +x /path/to/hive-prop.sh), then run it this
>>way:
>>
>> /path/to/hive-prop.sh hive.metastore.warehouse.dir
>>
>> Back to asking for for metadata for a table. The following script will
>> determine the location of a particular partition for an external
>> "mydatabase.stocks" table:
>>
>> #!/bin/sh
>> hive -e "describe formatted mydatabase.stocks
>>partition(exchange='NASDAQ',
>> symbol='AAPL');" 2> /dev/null | grep Location | sed -e "s/Location:[
>>\t]*//"
>>
>> dean
>>
>> On Mon, Feb 11, 2013 at 4:59 PM, Parag Sarda <[EMAIL PROTECTED]>
>>wrote:
>>>
>>> Hello Hive Users,
>>>
>>> I am writing a program in java which is bundled as JAR and executed
>>>using
>>> hadoop jar command. I would like to access hive metadata (read
>>>partitions
>>> informations) in this program. I can ask user to set HIVE_CONF_DIR
>>> environment variable before calling my program or ask for any
>>>reasonable
>>> parameters to be passed. I do not want to force user to run hive
>>>megastore
>>> service if possible to increase reliability of program by avoiding
>>> external dependencies.
>>>
>>> What is the recommended way to get partitions information? Here is my
>>> understanding
>>> 1. Make sure my jar is bundled with hive-metastore[1] library.
>>> 2. Use HiveMetastoreClient[2]
>>>
>>> Is this correct? If yes, how to read the hive configuration[3] from
>>> HIVE_CONF_DIR?
>>>
>>> [1] http://mvnrepository.com/artifact/org.apache.hive/hive-metastore
>>> [2]
>>>
>>>
>>>http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/metastore/
>>>Hiv
>>> eMetaStoreClient.html
>>> [3]
>>>
>>>
>>>http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/conf/HiveC
>>>onf
>>> .html
>>>
>>> Thanks in advance,
>>> Parag
>>>
>>
>>
>>
>> --
>> Dean Wampler, Ph.D.
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
+
Dean Wampler 2013-02-12, 16:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB