Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> How to load hive metadata from conf dir


Copy link to this message
-
Re: How to load hive metadata from conf dir
I'll mention another bash hack that I use all the time:

hive -e 'some_command' | grep for_what_i_want |
sed_command_to_remove_just_i_dont_want

For example, the following command will print just the value of
hive.metastore.warehouse.dir, sending all the logging junk written to
stderr to /dev/null and stripping off the leading
"hive.metastore.warehouse.dir=" from the stdout output:

hive -e 'set hive.metastore.warehouse.dir;' 2> /dev/null | sed -e
's/hive.metastore.warehouse.dir=//'

(No grep subcommand required in this case...)

You could do something similar with DESCRIBE EXTENDED table PARTION(...)
Suppose you want a script that works for any property. Put the following in
a script file, say hive-prop.sh:

#!/bin/sh
hive -e "set $1;" 2> /dev/null | sed -e "s/$1=//"

Make it executable (chmod +x /path/to/hive-prop.sh), then run it this way:

/path/to/hive-prop.sh hive.metastore.warehouse.dir

Back to asking for for metadata for a table. The following script will
determine the location of a particular partition for an external
"mydatabase.stocks" table:

#!/bin/sh
hive -e "describe formatted mydatabase.stocks partition(exchange='NASDAQ',
symbol='AAPL');" 2> /dev/null | grep Location | sed -e "s/Location:[ \t]*//"

dean

On Mon, Feb 11, 2013 at 4:59 PM, Parag Sarda <[EMAIL PROTECTED]> wrote:

> Hello Hive Users,
>
> I am writing a program in java which is bundled as JAR and executed using
> hadoop jar command. I would like to access hive metadata (read partitions
> informations) in this program. I can ask user to set HIVE_CONF_DIR
> environment variable before calling my program or ask for any reasonable
> parameters to be passed. I do not want to force user to run hive megastore
> service if possible to increase reliability of program by avoiding
> external dependencies.
>
> What is the recommended way to get partitions information? Here is my
> understanding
> 1. Make sure my jar is bundled with hive-metastore[1] library.
> 2. Use HiveMetastoreClient[2]
>
> Is this correct? If yes, how to read the hive configuration[3] from
> HIVE_CONF_DIR?
>
> [1] http://mvnrepository.com/artifact/org.apache.hive/hive-metastore
> [2]
> http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/metastore/Hiv
> eMetaStoreClient.html
> [3]
> http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/conf/HiveConf
> .html
>
> Thanks in advance,
> Parag
>
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330