Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # dev >> non-string partition columns


+
Namit Jain 2012-05-25, 01:49
+
Namit Jain 2012-05-25, 05:59
+
Ashutosh Chauhan 2012-05-25, 06:21
+
Namit Jain 2012-05-25, 17:01
+
Namit Jain 2012-05-26, 05:55
+
Philip Tromans 2012-05-26, 08:20
+
Aniket Mokashi 2012-05-29, 08:10
+
Edward Capriolo 2012-05-29, 16:27
+
Namit Jain 2012-05-29, 17:02
Copy link to this message
-
Re: non-string partition columns
FWIW.. HCatalog only allows partition columns of type string precisely
because in backend datastore type information is not recorded. In my
opinion, partition type should be restricted to type string until we fix
this problem, otherwise it gives unexpected behavior to endusers and/or
bug-reps. One possibility is to introduce config variable
hive.partition.column.type and has it value set to "string" by default.
This ensures that new users get expected behavior of string-only partition
columns. Users who already use other types can reset this config value to
"all" in their deployment when they upgrade to newer version of Hive
(assuming new version comes out without a proper fix). This extra step of
reseting default config will help them to understand the risk they are
taking by changing default value.

Thanks,
Ashutosh

On Tue, May 29, 2012 at 10:02 AM, Namit Jain <[EMAIL PROTECTED]> wrote:

> OK, I will keep the support.
> Add special casing for string columns in DDLTask
>
> On 5/29/12 9:27 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:
>
> >We use them to we store our dates as integers like 20120130. This
> >allows us to do partition pruning with ranges.
> >
> >On Tue, May 29, 2012 at 4:10 AM, Aniket Mokashi <[EMAIL PROTECTED]>
> >wrote:
> >> We are using non-string partition columns in production as well.
> >>
> >> Thanks,
> >> Aniket
> >>
> >> On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
> >> <[EMAIL PROTECTED]>wrote:
> >>
> >>> We're using non-string partition columns in production. I think non
> >>>string
> >>> partition columns are a good thing to have - it allows you to do all
> >>>sorts
> >>> of date range calculations etc. AFAIK, MySQL's partition columns can
> >>>be of
> >>> any type.
> >>>
> >>> Phil.
> >>> On May 26, 2012 7:55 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >>>
> >>> > Should I go ahead and file a jira to disallow non-string partition
> >>> columns
> >>> > ?
> >>> > Or, someone depends on that functionality.
> >>> >
> >>> >
> >>> > On 5/25/12 10:01 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >>> >
> >>> > >Yes, but the meta-question is:
> >>> > >
> >>> > >Is anyone dependent on non-string partition columns ? Should we
> >>>drop the
> >>> > >support for non-string
> >>> > >partition columns ?
> >>> > >
> >>> > >
> >>> > >Thanks,
> >>> > >-namit
> >>> > >
> >>> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]>
> >>>wrote:
> >>> > >
> >>> > >>Some discussion for this has happened on
> >>> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
> >>> > problem
> >>> > >>same as the one which I described on that jira ?
> >>> > >>
> >>> > >>Thanks,
> >>> > >>Ashutosh
> >>> > >>
> >>> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
> >>> > >>
> >>> > >>> Should we disallow non-string partition columns completely ?
> >>> > >>> Does anyone depend on that ?
> >>> > >>>
> >>> > >>>
> >>> > >>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >>> > >>>
> >>> > >>> >
> >>> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> >>> > >>> >
> >>> > >>> >The patch above broke drop partitions if the partition happens
> >>>to be
> >>> > >>> >non-string.
> >>> > >>> >This is due to a JDO issue with non-string columns.
> >>> > >>> >
> >>> > >>> >Is anyone using non-string partition columns ?
> >>> > >>> >Should be force the partition columns to be only of type string
> >>>?
> >>> > >>> >The documentation probably does not specify anything clearly.
> >>> > >>> >
> >>> > >>> >If someone is dependent on non-string partition column, we need
> >>>to
> >>> > >>>revert
> >>> > >>> >this patch, or make a
> >>> > >>> >special case for string partition columns.
> >>> > >>> >
> >>> > >>> >Thanks,
> >>> > >>> >-namit
> >>> > >>> >
> >>> > >>> >
> >>> > >>>
> >>> > >>>
> >>> > >
> >>> >
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> "...:::Aniket:::... Quetzalco@tl"
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB