Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - dynamic Partition not splitting properly


Copy link to this message
-
Re: dynamic Partition not splitting properly
Nitin Pawar 2013-06-13, 11:12
you can partition existing table unless the hdfs data is laid out in
partitioned fashion.
your best bet is create a new partitioned table
enable dynamic paritionining
read from old table and write into new table

you can verify the new partitions by using command "show partitions"
On Thu, Jun 13, 2013 at 4:40 PM, Hamza Asad <[EMAIL PROTECTED]> wrote:

> now i created partition table like
> *CREATE TABLE new_rc_partition_cluster_table(
>
>   id int,
>   event_id int,
>   user_id BIGINT,
>
>   intval_1 int ,
>   intval_2 int,
>   intval_3 int,
>   intval_4 int,
>   intval_5 int,
>   intval_6 int,
>   intval_7 int,
>   intval_8 int,
>   intval_9 int,
>   intval_10 int,
>   intval_11 int,
>   intval_12 int,
>   intval_13 int,
>   intval_14 int,
>   intval_15 int,
>   intval_16 int,
>   intval_17 int,
>   intval_18 int,
>   intval_19 int,
>   intval_20 int,
>   intval_21 int,
>   intval_22 int,
>   intval_23 int,
>   intval_24 int,
>   intval_25 int,
>   intval_26 int)
>   PARTITIONED BY (event_date string)
>
> CLUSTERED BY(id) INTO 256 BUCKETS
> ROW FORMAT DELIMITED
>  FIELDS TERMINATED BY ','
> STORED AS RCFile;
> *
>
> rest of the commands are same. But this time INSERT OVERWRITE query
> executed and took time but when i queries from that table, it results none
> as it does not contains data. why is this so? and also please tell me how
> can i partition my existing table dynamicaly on date so that data splits
> equally without mentioning date explicitly?
>
>
> On Wed, Jun 12, 2013 at 6:50 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> you did not create partitioned table. You just created a bucketed table.
>>
>> refer to partitioned table created
>> something like
>> partitioned by (event_date string)
>>
>>
>> On Wed, Jun 12, 2013 at 7:17 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>
>>> i have created table after enabling dynamic partition. i partitioned it
>>> on date but it is not splitting data datewise. Below is the query of table
>>> created and data insert
>>> CREATE TABLE rc_partition_cluster_table(
>>>   id int,
>>>   event_id int,
>>>   user_id BIGINT,
>>>   event_date string,
>>>   intval_1 int )
>>> CLUSTERED BY(id) INTO 256 BUCKETS
>>> ROW FORMAT DELIMITED
>>>  FIELDS TERMINATED BY ','
>>> STORED AS RCFile;
>>>
>>> set hive.exec.dynamic.partition=true;
>>> set hive.exec.dynamic.partition.mode=nonstrict;
>>> set hive.exec.max.dynamic.partitions=1000;
>>> set hive.exec.max.dynamic.partitions.pernode=1000;
>>>
>>> INSERT OVERWRITE TABLE rc_partition_cluster_table Partition (event_date)
>>> SELECT * FROM events_details;
>>>
>>> why it is not working fine?
>>>
>>> --
>>> *Muhammad Hamza Asad*
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> *Muhammad Hamza Asad*
>

--
Nitin Pawar