Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> dynamic Partition not splitting properly


Copy link to this message
-
Re: dynamic Partition not splitting properly
ok .. got it.. Thanx :)
p.s Nitin, have u any idea of indexes? i have emailed issue with
subject "*index
not working*", can u see it please?
On Fri, Jun 14, 2013 at 1:30 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> thats a wrong query
>
> insert into table table_name partition (partition_names) select cols,
> do_data(event_date) from table
>
> this is how it should look like
> hive will take care of inserting into respective partitions after you
> enable dynamic partitions
>
>
> On Fri, Jun 14, 2013 at 1:21 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>
>> i 'm executing following command but it fail to recognize partition column
>> *INSERT INTO TABLE rc_partition_table
>> PARTITION (to_date(event_date))
>> SELECT * FROM events_details*
>>
>>
>> On Fri, Jun 14, 2013 at 12:41 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>
>>> just use hive split function for strings and get the value extracted.
>>> by the way I am not sure why the to_date function is failing
>>> stringto_date(string timestamp)Returns the date part of a timestamp
>>> string: to_date("1970-01-01 00:00:00") = "1970-01-01"
>>> also, I am sorry i might have misread your question
>>> what do you mean by "partition column does not accepts
>>> to_date(event_date) form "
>>>
>>>
>>>
>>> On Fri, Jun 14, 2013 at 1:04 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>
>>>> sample row of my data is
>>>> *591269735,1,1022,2012-06-24
>>>> 11:08:10.9,null,2,null,null,null,null,null,null,null,null,12,null,null,2,null,null,null,null,4,1,null,null,null,null,null,null
>>>> *
>>>>
>>>> and i want to partition it according to date i.e *2012-06-24 *skiping
>>>> the hh:mm:ss.ff part
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 12:27 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> can you provide whats your data and what you want it to look like ?
>>>>>
>>>>>
>>>>> On Fri, Jun 14, 2013 at 12:31 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> which UDF? it does not take to_date(event_date) column
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 14, 2013 at 11:54 AM, Nitin Pawar <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> use already existing UDFs to split or transform your values the way
>>>>>>> you want
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 14, 2013 at 12:09 PM, Hamza Asad <[EMAIL PROTECTED]
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> OIC. I got it. Thanx alot nitin :). One more thing i want to ask
>>>>>>>> related this issue, if old table contains event_date in format "2012-06-24
>>>>>>>> 06:04:11.9" then how can i partition it according to date part only? As
>>>>>>>> partition column does not accepts to_date(event_date) form.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 13, 2013 at 5:07 PM, Nitin Pawar <
>>>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>>>
>>>>>>>>> If the input column value is NULL or empty string, the row will be put into a special partition, whose name is controlled by the hive parameter hive.exec.default.dynamic.partition.name. The default value is `__HIVE_DEFAULT_PARTITION__`. Basically this partition will contain all
>>>>>>>>> "bad" rows whose value are not valid partition names.
>>>>>>>>>
>>>>>>>>> so basically you do following things
>>>>>>>>>
>>>>>>>>> when you create a partitioned table, your partitioned column is normally at the end of the table, so when you are inserting data into this partitioned table, I would recommend using the column names in place select * from
>>>>>>>>>
>>>>>>>>> so your insert query should look like
>>>>>>>>>
>>>>>>>>> set hive.exec.dynamic.partition=true;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> set hive.exec.dynamic.partition.mode=nonstrict;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> insert overwrite table new_table partition(event_date) select col1, col2 .... coln, event_date from old_table;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 13, 2013 at 5:24 PM, Hamza Asad <
>>>>>>>>> [EMAIL PROTECTED]> wrote:

*Muhammad Hamza Asad*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB