Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - dynamic Partition not splitting properly


+
Hamza Asad 2013-06-12, 13:47
+
Nitin Pawar 2013-06-12, 13:50
+
Hamza Asad 2013-06-13, 11:10
+
Nitin Pawar 2013-06-13, 11:12
+
Hamza Asad 2013-06-13, 11:49
Copy link to this message
-
Re: dynamic Partition not splitting properly
Nitin Pawar 2013-06-13, 11:52
what do you mean when you say "it wont split correctly" ?
On Thu, Jun 13, 2013 at 5:19 PM, Hamza Asad <[EMAIL PROTECTED]> wrote:

> what if i have data of more then 500 days then how can i create partition
> on date column by specifying each and every date? (i knw that does not
> happens in dynamic partition but on dynamic partition, it wont splits
> correctly).
>
>
> On Thu, Jun 13, 2013 at 4:12 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> you can partition existing table unless the hdfs data is laid out in
>> partitioned fashion.
>> your best bet is create a new partitioned table
>> enable dynamic paritionining
>> read from old table and write into new table
>>
>> you can verify the new partitions by using command "show partitions"
>>
>>
>> On Thu, Jun 13, 2013 at 4:40 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>
>>> now i created partition table like
>>> *CREATE TABLE new_rc_partition_cluster_table(
>>>
>>>   id int,
>>>   event_id int,
>>>   user_id BIGINT,
>>>
>>>   intval_1 int ,
>>>   intval_2 int,
>>>   intval_3 int,
>>>   intval_4 int,
>>>   intval_5 int,
>>>   intval_6 int,
>>>   intval_7 int,
>>>   intval_8 int,
>>>   intval_9 int,
>>>   intval_10 int,
>>>   intval_11 int,
>>>   intval_12 int,
>>>   intval_13 int,
>>>   intval_14 int,
>>>   intval_15 int,
>>>   intval_16 int,
>>>   intval_17 int,
>>>   intval_18 int,
>>>   intval_19 int,
>>>   intval_20 int,
>>>   intval_21 int,
>>>   intval_22 int,
>>>   intval_23 int,
>>>   intval_24 int,
>>>   intval_25 int,
>>>   intval_26 int)
>>>   PARTITIONED BY (event_date string)
>>>
>>> CLUSTERED BY(id) INTO 256 BUCKETS
>>> ROW FORMAT DELIMITED
>>>  FIELDS TERMINATED BY ','
>>> STORED AS RCFile;
>>> *
>>>
>>> rest of the commands are same. But this time INSERT OVERWRITE query
>>> executed and took time but when i queries from that table, it results none
>>> as it does not contains data. why is this so? and also please tell me how
>>> can i partition my existing table dynamicaly on date so that data splits
>>> equally without mentioning date explicitly?
>>>
>>>
>>> On Wed, Jun 12, 2013 at 6:50 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>
>>>> you did not create partitioned table. You just created a bucketed table.
>>>>
>>>> refer to partitioned table created
>>>> something like
>>>> partitioned by (event_date string)
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 7:17 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> i have created table after enabling dynamic partition. i partitioned
>>>>> it on date but it is not splitting data datewise. Below is the query of
>>>>> table created and data insert
>>>>> CREATE TABLE rc_partition_cluster_table(
>>>>>   id int,
>>>>>   event_id int,
>>>>>   user_id BIGINT,
>>>>>   event_date string,
>>>>>   intval_1 int )
>>>>> CLUSTERED BY(id) INTO 256 BUCKETS
>>>>> ROW FORMAT DELIMITED
>>>>>  FIELDS TERMINATED BY ','
>>>>> STORED AS RCFile;
>>>>>
>>>>> set hive.exec.dynamic.partition=true;
>>>>> set hive.exec.dynamic.partition.mode=nonstrict;
>>>>> set hive.exec.max.dynamic.partitions=1000;
>>>>> set hive.exec.max.dynamic.partitions.pernode=1000;
>>>>>
>>>>> INSERT OVERWRITE TABLE rc_partition_cluster_table Partition
>>>>> (event_date)
>>>>> SELECT * FROM events_details;
>>>>>
>>>>> why it is not working fine?
>>>>>
>>>>> --
>>>>> *Muhammad Hamza Asad*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>>
>>> --
>>> *Muhammad Hamza Asad*
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> *Muhammad Hamza Asad*
>

--
Nitin Pawar
+
Hamza Asad 2013-06-13, 11:54
+
Nitin Pawar 2013-06-13, 12:07
+
Hamza Asad 2013-06-14, 06:39
+
Nitin Pawar 2013-06-14, 06:54
+
Hamza Asad 2013-06-14, 07:01
+
Nitin Pawar 2013-06-14, 07:27
+
Hamza Asad 2013-06-14, 07:34
+
Nitin Pawar 2013-06-14, 07:41
+
Hamza Asad 2013-06-14, 07:51
+
Nitin Pawar 2013-06-14, 08:30
+
Hamza Asad 2013-06-14, 08:38
+
Hamza Asad 2013-06-16, 13:08