Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Parallel Load Data into Two partitions of a Hive Table


Copy link to this message
-
Re: Parallel Load Data into Two partitions of a Hive Table
load data to different partitions parallel is OK, because it equivalent to
write to different file on HDFS
2013/5/3 selva <[EMAIL PROTECTED]>

> Hi All,
>
> I need to load a month worth of processed data into a hive table. Table
> have 10 partitions. Each day have many files to load and each file is
> taking two seconds(constantly) and i have ~3000 files). So it will take
> days to complete for 30 days worth of data.
>
> I planned to load every day data parallel into respective partition so
> that i can complete it short time.
>
> But i need clarrification before proceeding it.
>
> Question:
>
> 1. Will it cause data loss/corruption by loading parallel in different
> partition of same hive table ?
>
> For example, Assume i am doing like below,
>
> Table     : processedlogs
> Partition : logdate
>
> Running below commands parallel,
> LOAD DATA INPATH '/logs/processed/2013-04-01' OVERWRITE INTO TABLE
> processedlogs PARTITION(logdate='2013-04-01');
> LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE
> processedlogs PARTITION(logdate='2013-04-02');
> LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE
> processedlogs PARTITION(logdate='2013-04-03');
> LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE
> processedlogs PARTITION(logdate='2013-04-04');
> .....
> LOAD DATA INPATH '/logs/processed/2013-04-30' OVERWRITE INTO TABLE
> processedlogs PARTITION(logdate='2013-04-30');
>
> Thanks
> Selva
>
>
>
>
>
>