Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - dynamic Partition not splitting properly


Copy link to this message
-
Re: dynamic Partition not splitting properly
Hamza Asad 2013-06-13, 11:54
when i browse it in browser, all the data is in *
event_date=__HIVE_DEFAULT_PARTITION__<http://10.0.0.14:50075/browseDirectory.jsp?dir=%2Fvar%2Flog%2Fpring%2Fhive%2Fwarehouse%2Fnydus.db%2Fnew_rc_partition_cluster_table%2Fevent_date%3D__HIVE_DEFAULT_PARTITION__&namenodeInfoPort=50070>
*, rest of the files does not contains data
On Thu, Jun 13, 2013 at 4:52 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> what do you mean when you say "it wont split correctly" ?
>
>
> On Thu, Jun 13, 2013 at 5:19 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>
>> what if i have data of more then 500 days then how can i create partition
>> on date column by specifying each and every date? (i knw that does not
>> happens in dynamic partition but on dynamic partition, it wont splits
>> correctly).
>>
>>
>> On Thu, Jun 13, 2013 at 4:12 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>
>>> you can partition existing table unless the hdfs data is laid out in
>>> partitioned fashion.
>>> your best bet is create a new partitioned table
>>> enable dynamic paritionining
>>> read from old table and write into new table
>>>
>>> you can verify the new partitions by using command "show partitions"
>>>
>>>
>>> On Thu, Jun 13, 2013 at 4:40 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>
>>>> now i created partition table like
>>>> *CREATE TABLE new_rc_partition_cluster_table(
>>>>
>>>>   id int,
>>>>   event_id int,
>>>>   user_id BIGINT,
>>>>
>>>>   intval_1 int ,
>>>>   intval_2 int,
>>>>   intval_3 int,
>>>>   intval_4 int,
>>>>   intval_5 int,
>>>>   intval_6 int,
>>>>   intval_7 int,
>>>>   intval_8 int,
>>>>   intval_9 int,
>>>>   intval_10 int,
>>>>   intval_11 int,
>>>>   intval_12 int,
>>>>   intval_13 int,
>>>>   intval_14 int,
>>>>   intval_15 int,
>>>>   intval_16 int,
>>>>   intval_17 int,
>>>>   intval_18 int,
>>>>   intval_19 int,
>>>>   intval_20 int,
>>>>   intval_21 int,
>>>>   intval_22 int,
>>>>   intval_23 int,
>>>>   intval_24 int,
>>>>   intval_25 int,
>>>>   intval_26 int)
>>>>   PARTITIONED BY (event_date string)
>>>>
>>>> CLUSTERED BY(id) INTO 256 BUCKETS
>>>> ROW FORMAT DELIMITED
>>>>  FIELDS TERMINATED BY ','
>>>> STORED AS RCFile;
>>>> *
>>>>
>>>> rest of the commands are same. But this time INSERT OVERWRITE query
>>>> executed and took time but when i queries from that table, it results none
>>>> as it does not contains data. why is this so? and also please tell me how
>>>> can i partition my existing table dynamicaly on date so that data splits
>>>> equally without mentioning date explicitly?
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 6:50 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> you did not create partitioned table. You just created a bucketed
>>>>> table.
>>>>>
>>>>> refer to partitioned table created
>>>>> something like
>>>>> partitioned by (event_date string)
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 7:17 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> i have created table after enabling dynamic partition. i partitioned
>>>>>> it on date but it is not splitting data datewise. Below is the query of
>>>>>> table created and data insert
>>>>>> CREATE TABLE rc_partition_cluster_table(
>>>>>>   id int,
>>>>>>   event_id int,
>>>>>>   user_id BIGINT,
>>>>>>   event_date string,
>>>>>>   intval_1 int )
>>>>>> CLUSTERED BY(id) INTO 256 BUCKETS
>>>>>> ROW FORMAT DELIMITED
>>>>>>  FIELDS TERMINATED BY ','
>>>>>> STORED AS RCFile;
>>>>>>
>>>>>> set hive.exec.dynamic.partition=true;
>>>>>> set hive.exec.dynamic.partition.mode=nonstrict;
>>>>>> set hive.exec.max.dynamic.partitions=1000;
>>>>>> set hive.exec.max.dynamic.partitions.pernode=1000;
>>>>>>
>>>>>> INSERT OVERWRITE TABLE rc_partition_cluster_table Partition
>>>>>> (event_date)
>>>>>> SELECT * FROM events_details;
>>>>>>
>>>>>> why it is not working fine?
>>>>>>
>>>>>> --
>>>>>> *Muhammad Hamza Asad*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nitin Pawar
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Muhammad Hamza Asad*
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>

*Muhammad Hamza Asad*