Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Parallely Load Data into Two partitions of a Hive Table


+
selva 2013-05-03, 07:04
Copy link to this message
-
Re: Parallely Load Data into Two partitions of a Hive Table
Why dont you load all of your data into a temporary table and then from
there to your current tables.

Hive will take care of adding dynic partitions and that will remove  the
ocerhead from you.

To answer your question, you can always load data in different partitions
parallely as long as you have resources available on hive cli machine

On May 3, 2013 12:35 PM, "selva" <[EMAIL PROTECTED]> wrote:
>
> Hi All,
>
> I need to load a month worth of processed data into a hive table. Table
have 10 partitions. Each day have many files to load and each file is
taking two seconds(constantly) and i have ~3000 files). So it will take
days to complete for 30 days worth of data.
>
> I planned to load every day data parellaly into respective partition so
that i can complete it short time.
>
> But i need clarrification before proceeding it.
>
> Question:
>
> 1. Will it cause data loss/corruption by loading parellely in different
partition of same hive table ?
>
> For example, Assume i am doing like below,
>
> Table : processedlogs
> Partition : logdate
>
> Running below commands parellely,
> LOAD DATA INPATH '/logs/processed/2013-04-01' OVERWRITE INTO TABLE
processedlogs PARTITION(logdate='2013-04-01');
> LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE
processedlogs PARTITION(logdate='2013-04-02');
> LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE
processedlogs PARTITION(logdate='2013-04-03');
> LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE
processedlogs PARTITION(logdate='2013-04-04');
> .....
> LOAD DATA INPATH '/logs/processed/2013-04-30' OVERWRITE INTO TABLE
processedlogs PARTITION(logdate='2013-04-30');
>
> Thanks
> Selva
>
>
>
>
>
>
>
+
selva 2013-05-03, 08:21
+
Sanjay Subramanian 2013-05-03, 08:42