Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> dynamic Partition not splitting properly


+
Hamza Asad 2013-06-12, 13:47
+
Nitin Pawar 2013-06-12, 13:50
+
Hamza Asad 2013-06-13, 11:10
+
Nitin Pawar 2013-06-13, 11:12
+
Hamza Asad 2013-06-13, 11:49
+
Nitin Pawar 2013-06-13, 11:52
+
Hamza Asad 2013-06-13, 11:54
+
Nitin Pawar 2013-06-13, 12:07
+
Hamza Asad 2013-06-14, 06:39
+
Nitin Pawar 2013-06-14, 06:54
+
Hamza Asad 2013-06-14, 07:01
+
Nitin Pawar 2013-06-14, 07:27
+
Hamza Asad 2013-06-14, 07:34
+
Nitin Pawar 2013-06-14, 07:41
+
Hamza Asad 2013-06-14, 07:51
Copy link to this message
-
Re: dynamic Partition not splitting properly
thats a wrong query

insert into table table_name partition (partition_names) select cols,
do_data(event_date) from table

this is how it should look like
hive will take care of inserting into respective partitions after you
enable dynamic partitions
On Fri, Jun 14, 2013 at 1:21 PM, Hamza Asad <[EMAIL PROTECTED]> wrote:

> i 'm executing following command but it fail to recognize partition column
> *INSERT INTO TABLE rc_partition_table
> PARTITION (to_date(event_date))
> SELECT * FROM events_details*
>
>
> On Fri, Jun 14, 2013 at 12:41 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> just use hive split function for strings and get the value extracted.
>> by the way I am not sure why the to_date function is failing
>> stringto_date(string timestamp)Returns the date part of a timestamp
>> string: to_date("1970-01-01 00:00:00") = "1970-01-01"
>> also, I am sorry i might have misread your question
>> what do you mean by "partition column does not accepts
>> to_date(event_date) form "
>>
>>
>>
>> On Fri, Jun 14, 2013 at 1:04 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>
>>> sample row of my data is
>>> *591269735,1,1022,2012-06-24
>>> 11:08:10.9,null,2,null,null,null,null,null,null,null,null,12,null,null,2,null,null,null,null,4,1,null,null,null,null,null,null
>>> *
>>>
>>> and i want to partition it according to date i.e *2012-06-24 *skiping
>>> the hh:mm:ss.ff part
>>>
>>>
>>> On Fri, Jun 14, 2013 at 12:27 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>
>>>> can you provide whats your data and what you want it to look like ?
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 12:31 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> which UDF? it does not take to_date(event_date) column
>>>>>
>>>>>
>>>>> On Fri, Jun 14, 2013 at 11:54 AM, Nitin Pawar <[EMAIL PROTECTED]
>>>>> > wrote:
>>>>>
>>>>>> use already existing UDFs to split or transform your values the way
>>>>>> you want
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 14, 2013 at 12:09 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> OIC. I got it. Thanx alot nitin :). One more thing i want to ask
>>>>>>> related this issue, if old table contains event_date in format "2012-06-24
>>>>>>> 06:04:11.9" then how can i partition it according to date part only? As
>>>>>>> partition column does not accepts to_date(event_date) form.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 13, 2013 at 5:07 PM, Nitin Pawar <
>>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>>
>>>>>>>> If the input column value is NULL or empty string, the row will be put into a special partition, whose name is controlled by the hive parameter hive.exec.default.dynamic.partition.name. The default value is `__HIVE_DEFAULT_PARTITION__`. Basically this partition will contain all
>>>>>>>> "bad" rows whose value are not valid partition names.
>>>>>>>>
>>>>>>>> so basically you do following things
>>>>>>>>
>>>>>>>> when you create a partitioned table, your partitioned column is normally at the end of the table, so when you are inserting data into this partitioned table, I would recommend using the column names in place select * from
>>>>>>>>
>>>>>>>> so your insert query should look like
>>>>>>>>
>>>>>>>> set hive.exec.dynamic.partition=true;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> set hive.exec.dynamic.partition.mode=nonstrict;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> insert overwrite table new_table partition(event_date) select col1, col2 .... coln, event_date from old_table;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 13, 2013 at 5:24 PM, Hamza Asad <[EMAIL PROTECTED]
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> when i browse it in browser, all the data is in *
>>>>>>>>> event_date=__HIVE_DEFAULT_PARTITION__<http://10.0.0.14:50075/browseDirectory.jsp?dir=%2Fvar%2Flog%2Fpring%2Fhive%2Fwarehouse%2Fnydus.db%2Fnew_rc_partition_cluster_table%2Fevent_date%3D__HIVE_DEFAULT_PARTITION__&namenodeInfoPort=50070>
>>>>>>>>> *, rest of the files does not contains data

Nitin Pawar
+
Hamza Asad 2013-06-14, 08:38
+
Hamza Asad 2013-06-16, 13:08
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB