Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> dynamic Partition not splitting properly


+
Hamza Asad 2013-06-12, 13:47
+
Nitin Pawar 2013-06-12, 13:50
+
Hamza Asad 2013-06-13, 11:10
+
Nitin Pawar 2013-06-13, 11:12
+
Hamza Asad 2013-06-13, 11:49
+
Nitin Pawar 2013-06-13, 11:52
+
Hamza Asad 2013-06-13, 11:54
+
Nitin Pawar 2013-06-13, 12:07
+
Hamza Asad 2013-06-14, 06:39
+
Nitin Pawar 2013-06-14, 06:54
+
Hamza Asad 2013-06-14, 07:01
+
Nitin Pawar 2013-06-14, 07:27
+
Hamza Asad 2013-06-14, 07:34
+
Nitin Pawar 2013-06-14, 07:41
+
Hamza Asad 2013-06-14, 07:51
+
Nitin Pawar 2013-06-14, 08:30
Copy link to this message
-
Re: dynamic Partition not splitting properly
ok .. got it.. Thanx :)
p.s Nitin, have u any idea of indexes? i have emailed issue with
subject "*index
not working*", can u see it please?
On Fri, Jun 14, 2013 at 1:30 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> thats a wrong query
>
> insert into table table_name partition (partition_names) select cols,
> do_data(event_date) from table
>
> this is how it should look like
> hive will take care of inserting into respective partitions after you
> enable dynamic partitions
>
>
> On Fri, Jun 14, 2013 at 1:21 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>
>> i 'm executing following command but it fail to recognize partition column
>> *INSERT INTO TABLE rc_partition_table
>> PARTITION (to_date(event_date))
>> SELECT * FROM events_details*
>>
>>
>> On Fri, Jun 14, 2013 at 12:41 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>
>>> just use hive split function for strings and get the value extracted.
>>> by the way I am not sure why the to_date function is failing
>>> stringto_date(string timestamp)Returns the date part of a timestamp
>>> string: to_date("1970-01-01 00:00:00") = "1970-01-01"
>>> also, I am sorry i might have misread your question
>>> what do you mean by "partition column does not accepts
>>> to_date(event_date) form "
>>>
>>>
>>>
>>> On Fri, Jun 14, 2013 at 1:04 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>
>>>> sample row of my data is
>>>> *591269735,1,1022,2012-06-24
>>>> 11:08:10.9,null,2,null,null,null,null,null,null,null,null,12,null,null,2,null,null,null,null,4,1,null,null,null,null,null,null
>>>> *
>>>>
>>>> and i want to partition it according to date i.e *2012-06-24 *skiping
>>>> the hh:mm:ss.ff part
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 12:27 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> can you provide whats your data and what you want it to look like ?
>>>>>
>>>>>
>>>>> On Fri, Jun 14, 2013 at 12:31 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> which UDF? it does not take to_date(event_date) column
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 14, 2013 at 11:54 AM, Nitin Pawar <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> use already existing UDFs to split or transform your values the way
>>>>>>> you want
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 14, 2013 at 12:09 PM, Hamza Asad <[EMAIL PROTECTED]
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> OIC. I got it. Thanx alot nitin :). One more thing i want to ask
>>>>>>>> related this issue, if old table contains event_date in format "2012-06-24
>>>>>>>> 06:04:11.9" then how can i partition it according to date part only? As
>>>>>>>> partition column does not accepts to_date(event_date) form.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 13, 2013 at 5:07 PM, Nitin Pawar <
>>>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>>>
>>>>>>>>> If the input column value is NULL or empty string, the row will be put into a special partition, whose name is controlled by the hive parameter hive.exec.default.dynamic.partition.name. The default value is `__HIVE_DEFAULT_PARTITION__`. Basically this partition will contain all
>>>>>>>>> "bad" rows whose value are not valid partition names.
>>>>>>>>>
>>>>>>>>> so basically you do following things
>>>>>>>>>
>>>>>>>>> when you create a partitioned table, your partitioned column is normally at the end of the table, so when you are inserting data into this partitioned table, I would recommend using the column names in place select * from
>>>>>>>>>
>>>>>>>>> so your insert query should look like
>>>>>>>>>
>>>>>>>>> set hive.exec.dynamic.partition=true;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> set hive.exec.dynamic.partition.mode=nonstrict;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> insert overwrite table new_table partition(event_date) select col1, col2 .... coln, event_date from old_table;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 13, 2013 at 5:24 PM, Hamza Asad <
>>>>>>>>> [EMAIL PROTECTED]> wrote:

*Muhammad Hamza Asad*
+
Hamza Asad 2013-06-16, 13:08