Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> question about partition table in hive


Copy link to this message
-
Re: question about partition table in hive
Flume might be able to invoke Hive to do this as the data is ingested, but
I don't know anything about Flume.

Brent has a nice blog post describing many of the details of partitioning.

http://www.brentozar.com/archive/2013/03/introduction-to-hive-partitioning/

We also cover them in our book. The key steps to taking the file(s) you
created and transforming them into partitioned data are the following:

1. Create an "external" table where the location is the directory you wrote
that big HDFS file (or files).
2. Create the final target table with the partitioning, as described in
Brent's blog post.
3. Run a query against the first table to populate the second. Again, Brent
covers the details.

See the Hive wiki for additional details on external tables, etc.

Dean

On Thu, Sep 12, 2013 at 7:55 PM, ch huang <[EMAIL PROTECTED]> wrote:

> hi,all:
>         i use flume collect log data and put it in hdfs ,i want to use
> hive to do some caculate, query based on timerange,i want to use parttion
> table ,
> but the data file in hdfs is a big file ,how can i put it into pratition
> table in hive?
>

--
Dean Wampler, Ph.D.
@deanwampler
http://polyglotprogramming.com