Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - How to load hive metadata from conf dir


+
Parag Sarda 2013-02-11, 22:59
+
Mark Grover 2013-02-12, 04:57
+
Parag Sarda 2013-02-12, 08:18
+
Nitin Pawar 2013-02-12, 08:25
+
Parag Sarda 2013-02-12, 16:36
+
Dean Wampler 2013-02-12, 14:41
+
Edward Capriolo 2013-02-12, 16:53
Copy link to this message
-
Re: How to load hive metadata from conf dir
Parag Sarda 2013-02-12, 17:31
Hive-thrift is definitely best option till now. That said, I am wondering
if its possible to load megastore in local mode[1] to avoid dependency on
external service. Can I read the HIVE_CONF_DIR for javax.jdo.option.*
parameters and talk to sql server hosting hive metadata?

[1] https://cwiki.apache.org/Hive/adminmanual-metastoreadmin.html

Thanks,
Parag
On 12/02/13 10:23 PM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:

>If you use hive-thrift/hive-service you can get the location of a
>table through the Table API (instead of Dean's horrid bash-isms)
>
>http://hive.apache.org/docs/r0.7.0/api/org/apache/hadoop/hive/metastore/ap
>i/ThriftHiveMetastore.Client.html#get_table(java.lang.String,
>java.lang.String)
>
>Table t = ....
>t.getSd().getLocation()
>
>
>On Tue, Feb 12, 2013 at 9:41 AM, Dean Wampler
><[EMAIL PROTECTED]> wrote:
>> I'll mention another bash hack that I use all the time:
>>
>> hive -e 'some_command' | grep for_what_i_want |
>> sed_command_to_remove_just_i_dont_want
>>
>> For example, the following command will print just the value of
>> hive.metastore.warehouse.dir, sending all the logging junk written to
>>stderr
>> to /dev/null and stripping off the leading
>>"hive.metastore.warehouse.dir="
>> from the stdout output:
>>
>> hive -e 'set hive.metastore.warehouse.dir;' 2> /dev/null | sed -e
>> 's/hive.metastore.warehouse.dir=//'
>>
>> (No grep subcommand required in this case...)
>>
>> You could do something similar with DESCRIBE EXTENDED table PARTION(...)
>> Suppose you want a script that works for any property. Put the
>>following in
>> a script file, say hive-prop.sh:
>>
>> #!/bin/sh
>> hive -e "set $1;" 2> /dev/null | sed -e "s/$1=//"
>>
>> Make it executable (chmod +x /path/to/hive-prop.sh), then run it this
>>way:
>>
>> /path/to/hive-prop.sh hive.metastore.warehouse.dir
>>
>> Back to asking for for metadata for a table. The following script will
>> determine the location of a particular partition for an external
>> "mydatabase.stocks" table:
>>
>> #!/bin/sh
>> hive -e "describe formatted mydatabase.stocks
>>partition(exchange='NASDAQ',
>> symbol='AAPL');" 2> /dev/null | grep Location | sed -e "s/Location:[
>>\t]*//"
>>
>> dean
>>
>> On Mon, Feb 11, 2013 at 4:59 PM, Parag Sarda <[EMAIL PROTECTED]>
>>wrote:
>>>
>>> Hello Hive Users,
>>>
>>> I am writing a program in java which is bundled as JAR and executed
>>>using
>>> hadoop jar command. I would like to access hive metadata (read
>>>partitions
>>> informations) in this program. I can ask user to set HIVE_CONF_DIR
>>> environment variable before calling my program or ask for any
>>>reasonable
>>> parameters to be passed. I do not want to force user to run hive
>>>megastore
>>> service if possible to increase reliability of program by avoiding
>>> external dependencies.
>>>
>>> What is the recommended way to get partitions information? Here is my
>>> understanding
>>> 1. Make sure my jar is bundled with hive-metastore[1] library.
>>> 2. Use HiveMetastoreClient[2]
>>>
>>> Is this correct? If yes, how to read the hive configuration[3] from
>>> HIVE_CONF_DIR?
>>>
>>> [1] http://mvnrepository.com/artifact/org.apache.hive/hive-metastore
>>> [2]
>>>
>>>
>>>http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/metastore/
>>>Hiv
>>> eMetaStoreClient.html
>>> [3]
>>>
>>>
>>>http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/conf/HiveC
>>>onf
>>> .html
>>>
>>> Thanks in advance,
>>> Parag
>>>
>>
>>
>>
>> --
>> Dean Wampler, Ph.D.
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
+
Dean Wampler 2013-02-12, 16:55