Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Change in serdeproperties does not update existing partitions


Copy link to this message
-
Re: Change in serdeproperties does not update existing partitions
Hey Maxime,

Yeah, thats intended behavior. After you do alter on table, all subsequent
actions on table and partitions will inherit from it. If you want to modify
properties of already existing partitions, you should be able to do
something like 'alter table test_table partition (day='2011-09-02') set
serdeproperties ('input.regex' = '(.*)')' Unfortunately this is not
supported currently. Feel free to file a bug for that.

A workaround (applicable only because you are using external table) is to
drop partition and then add them again. When you drop a partition from
external table, only metadata gets wiped out, data is not deleted, so when
you will add partition again, it will inherit from table serde properties
and you will get what you are looking for. Use this workaround with care,
you don't want to loose your data in recreating partitions.

Hope it helps,
Ashutosh

On Tue, Sep 13, 2011 at 06:03, Maxime Brugidou <[EMAIL PROTECTED]>wrote:

> Hello,
>
> I am using Hive 0.7 from cloudera cdh3u0 and I encounter a strange behavior
> when I update the serdeproperties of a table (for example for the
> RegexSerDe).
>
> If you have a simple partitioned table like
>
> create external table test_table (
>     id int)
> partitioned by (day string)
> row format serde 'org.apache.hadoop.contrib.serde2.RegexSerDe'
> with serdeproperties (
>     'input.regex' = '.* ([^ ]*)'
> );
>
> alter table test_table add partition (day='2011-09-01');
>
> alter table test_table set serdeproperties  (
>     'input.regex' = '(.*)'
> );
>
> alter table test_table add partition (day='2011-09-02');
>
>
> The first partition will still use the older regex and the new one will use
> the new regex. Is this intended behavior? Why?
>
> Thanks for your help,
> Maxime
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB