Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - How to load hive metadata from conf dir


Copy link to this message
-
Re: How to load hive metadata from conf dir
Dean Wampler 2013-02-12, 16:55
But then you're writing Java code!!! The Horror!!!

;^P

On Tue, Feb 12, 2013 at 10:53 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> If you use hive-thrift/hive-service you can get the location of a
> table through the Table API (instead of Dean's horrid bash-isms)
>
>
> http://hive.apache.org/docs/r0.7.0/api/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.Client.html#get_table(java.lang.String
> ,
> java.lang.String)
>
> Table t = ....
> t.getSd().getLocation()
>
>
> On Tue, Feb 12, 2013 at 9:41 AM, Dean Wampler
> <[EMAIL PROTECTED]> wrote:
> > I'll mention another bash hack that I use all the time:
> >
> > hive -e 'some_command' | grep for_what_i_want |
> > sed_command_to_remove_just_i_dont_want
> >
> > For example, the following command will print just the value of
> > hive.metastore.warehouse.dir, sending all the logging junk written to
> stderr
> > to /dev/null and stripping off the leading
> "hive.metastore.warehouse.dir="
> > from the stdout output:
> >
> > hive -e 'set hive.metastore.warehouse.dir;' 2> /dev/null | sed -e
> > 's/hive.metastore.warehouse.dir=//'
> >
> > (No grep subcommand required in this case...)
> >
> > You could do something similar with DESCRIBE EXTENDED table PARTION(...)
> > Suppose you want a script that works for any property. Put the following
> in
> > a script file, say hive-prop.sh:
> >
> > #!/bin/sh
> > hive -e "set $1;" 2> /dev/null | sed -e "s/$1=//"
> >
> > Make it executable (chmod +x /path/to/hive-prop.sh), then run it this
> way:
> >
> > /path/to/hive-prop.sh hive.metastore.warehouse.dir
> >
> > Back to asking for for metadata for a table. The following script will
> > determine the location of a particular partition for an external
> > "mydatabase.stocks" table:
> >
> > #!/bin/sh
> > hive -e "describe formatted mydatabase.stocks
> partition(exchange='NASDAQ',
> > symbol='AAPL');" 2> /dev/null | grep Location | sed -e "s/Location:[
> \t]*//"
> >
> > dean
> >
> > On Mon, Feb 11, 2013 at 4:59 PM, Parag Sarda <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Hello Hive Users,
> >>
> >> I am writing a program in java which is bundled as JAR and executed
> using
> >> hadoop jar command. I would like to access hive metadata (read
> partitions
> >> informations) in this program. I can ask user to set HIVE_CONF_DIR
> >> environment variable before calling my program or ask for any reasonable
> >> parameters to be passed. I do not want to force user to run hive
> megastore
> >> service if possible to increase reliability of program by avoiding
> >> external dependencies.
> >>
> >> What is the recommended way to get partitions information? Here is my
> >> understanding
> >> 1. Make sure my jar is bundled with hive-metastore[1] library.
> >> 2. Use HiveMetastoreClient[2]
> >>
> >> Is this correct? If yes, how to read the hive configuration[3] from
> >> HIVE_CONF_DIR?
> >>
> >> [1] http://mvnrepository.com/artifact/org.apache.hive/hive-metastore
> >> [2]
> >>
> >>
> http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/metastore/Hiv
> >> eMetaStoreClient.html
> >> [3]
> >>
> >>
> http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/conf/HiveConf
> >> .html
> >>
> >> Thanks in advance,
> >> Parag
> >>
> >
> >
> >
> > --
> > Dean Wampler, Ph.D.
> > thinkbiganalytics.com
> > +1-312-339-1330
> >
>

--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330