http://svn.apache.org/viewvc?view=revision&revision=1308427

The patch above broke drop partitions if the partition happens to be non-string.
This is due to a JDO issue with non-string columns.

Is anyone using non-string partition columns ?
Should be force the partition columns to be only of type string ?
The documentation probably does not specify anything clearly.

If someone is dependent on non-string partition column, we need to revert this patch, or make a
special case for string partition columns.

Thanks,
-namit

Should we disallow non-string partition columns completely ?
Does anyone depend on that ?
On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:

>
>http://svn.apache.org/viewvc?view=revision&revision=1308427
>
>The patch above broke drop partitions if the partition happens to be
>non-string.
>This is due to a JDO issue with non-string columns.
>
>Is anyone using non-string partition columns ?
>Should be force the partition columns to be only of type string ?
>The documentation probably does not specify anything clearly.
>
>If someone is dependent on non-string partition column, we need to revert
>this patch, or make a
>special case for string partition columns.
>
>Thanks,
>-namit
>
>

Some discussion for this has happened on
https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying problem
same as the one which I described on that jira ?

Thanks,
Ashutosh

On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:

> Should we disallow non-string partition columns completely ?
> Does anyone depend on that ?
>
>
> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>
> >
> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> >
> >The patch above broke drop partitions if the partition happens to be
> >non-string.
> >This is due to a JDO issue with non-string columns.
> >
> >Is anyone using non-string partition columns ?
> >Should be force the partition columns to be only of type string ?
> >The documentation probably does not specify anything clearly.
> >
> >If someone is dependent on non-string partition column, we need to revert
> >this patch, or make a
> >special case for string partition columns.
> >
> >Thanks,
> >-namit
> >
> >
>
>

Yes, but the meta-question is:

Is anyone dependent on non-string partition columns ? Should we drop the
support for non-string
partition columns ?
Thanks,
-namit

On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]> wrote:

>Some discussion for this has happened on
>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying problem
>same as the one which I described on that jira ?
>
>Thanks,
>Ashutosh
>
>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
>
>> Should we disallow non-string partition columns completely ?
>> Does anyone depend on that ?
>>
>>
>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>>
>> >
>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>> >
>> >The patch above broke drop partitions if the partition happens to be
>> >non-string.
>> >This is due to a JDO issue with non-string columns.
>> >
>> >Is anyone using non-string partition columns ?
>> >Should be force the partition columns to be only of type string ?
>> >The documentation probably does not specify anything clearly.
>> >
>> >If someone is dependent on non-string partition column, we need to
>>revert
>> >this patch, or make a
>> >special case for string partition columns.
>> >
>> >Thanks,
>> >-namit
>> >
>> >
>>
>>

Should I go ahead and file a jira to disallow non-string partition columns
?
Or, someone depends on that functionality.
On 5/25/12 10:01 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:

>Yes, but the meta-question is:
>
>Is anyone dependent on non-string partition columns ? Should we drop the
>support for non-string
>partition columns ?
>
>
>Thanks,
>-namit
>
>On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]> wrote:
>
>>Some discussion for this has happened on
>>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying problem
>>same as the one which I described on that jira ?
>>
>>Thanks,
>>Ashutosh
>>
>>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
>>
>>> Should we disallow non-string partition columns completely ?
>>> Does anyone depend on that ?
>>>
>>>
>>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>>>
>>> >
>>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>>> >
>>> >The patch above broke drop partitions if the partition happens to be
>>> >non-string.
>>> >This is due to a JDO issue with non-string columns.
>>> >
>>> >Is anyone using non-string partition columns ?
>>> >Should be force the partition columns to be only of type string ?
>>> >The documentation probably does not specify anything clearly.
>>> >
>>> >If someone is dependent on non-string partition column, we need to
>>>revert
>>> >this patch, or make a
>>> >special case for string partition columns.
>>> >
>>> >Thanks,
>>> >-namit
>>> >
>>> >
>>>
>>>
>

We're using non-string partition columns in production. I think non string
partition columns are a good thing to have - it allows you to do all sorts
of date range calculations etc. AFAIK, MySQL's partition columns can be of
any type.

Phil.
On May 26, 2012 7:55 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:

> Should I go ahead and file a jira to disallow non-string partition columns
> ?
> Or, someone depends on that functionality.
>
>
> On 5/25/12 10:01 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>
> >Yes, but the meta-question is:
> >
> >Is anyone dependent on non-string partition columns ? Should we drop the
> >support for non-string
> >partition columns ?
> >
> >
> >Thanks,
> >-namit
> >
> >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]> wrote:
> >
> >>Some discussion for this has happened on
> >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
> problem
> >>same as the one which I described on that jira ?
> >>
> >>Thanks,
> >>Ashutosh
> >>
> >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
> >>
> >>> Should we disallow non-string partition columns completely ?
> >>> Does anyone depend on that ?
> >>>
> >>>
> >>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >>>
> >>> >
> >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> >>> >
> >>> >The patch above broke drop partitions if the partition happens to be
> >>> >non-string.
> >>> >This is due to a JDO issue with non-string columns.
> >>> >
> >>> >Is anyone using non-string partition columns ?
> >>> >Should be force the partition columns to be only of type string ?
> >>> >The documentation probably does not specify anything clearly.
> >>> >
> >>> >If someone is dependent on non-string partition column, we need to
> >>>revert
> >>> >this patch, or make a
> >>> >special case for string partition columns.
> >>> >
> >>> >Thanks,
> >>> >-namit
> >>> >
> >>> >
> >>>
> >>>
> >
>
>

We are using non-string partition columns in production as well.

Thanks,
Aniket

On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
<[EMAIL PROTECTED]>wrote:

> We're using non-string partition columns in production. I think non string
> partition columns are a good thing to have - it allows you to do all sorts
> of date range calculations etc. AFAIK, MySQL's partition columns can be of
> any type.
>
> Phil.
> On May 26, 2012 7:55 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>
> > Should I go ahead and file a jira to disallow non-string partition
> columns
> > ?
> > Or, someone depends on that functionality.
> >
> >
> > On 5/25/12 10:01 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >
> > >Yes, but the meta-question is:
> > >
> > >Is anyone dependent on non-string partition columns ? Should we drop the
> > >support for non-string
> > >partition columns ?
> > >
> > >
> > >Thanks,
> > >-namit
> > >
> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]> wrote:
> > >
> > >>Some discussion for this has happened on
> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
> > problem
> > >>same as the one which I described on that jira ?
> > >>
> > >>Thanks,
> > >>Ashutosh
> > >>
> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
> > >>
> > >>> Should we disallow non-string partition columns completely ?
> > >>> Does anyone depend on that ?
> > >>>
> > >>>
> > >>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> > >>>
> > >>> >
> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> > >>> >
> > >>> >The patch above broke drop partitions if the partition happens to be
> > >>> >non-string.
> > >>> >This is due to a JDO issue with non-string columns.
> > >>> >
> > >>> >Is anyone using non-string partition columns ?
> > >>> >Should be force the partition columns to be only of type string ?
> > >>> >The documentation probably does not specify anything clearly.
> > >>> >
> > >>> >If someone is dependent on non-string partition column, we need to
> > >>>revert
> > >>> >this patch, or make a
> > >>> >special case for string partition columns.
> > >>> >
> > >>> >Thanks,
> > >>> >-namit
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >
> >
> >
>

--
"...:::Aniket:::... Quetzalco@tl"

We use them to we store our dates as integers like 20120130. This
allows us to do partition pruning with ranges.

On Tue, May 29, 2012 at 4:10 AM, Aniket Mokashi <[EMAIL PROTECTED]> wrote:
> We are using non-string partition columns in production as well.
>
> Thanks,
> Aniket
>
> On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
> <[EMAIL PROTECTED]>wrote:
>
>> We're using non-string partition columns in production. I think non string
>> partition columns are a good thing to have - it allows you to do all sorts
>> of date range calculations etc. AFAIK, MySQL's partition columns can be of
>> any type.
>>
>> Phil.
>> On May 26, 2012 7:55 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>>
>> > Should I go ahead and file a jira to disallow non-string partition
>> columns
>> > ?
>> > Or, someone depends on that functionality.
>> >
>> >
>> > On 5/25/12 10:01 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>> >
>> > >Yes, but the meta-question is:
>> > >
>> > >Is anyone dependent on non-string partition columns ? Should we drop the
>> > >support for non-string
>> > >partition columns ?
>> > >
>> > >
>> > >Thanks,
>> > >-namit
>> > >
>> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]> wrote:
>> > >
>> > >>Some discussion for this has happened on
>> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
>> > problem
>> > >>same as the one which I described on that jira ?
>> > >>
>> > >>Thanks,
>> > >>Ashutosh
>> > >>
>> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
>> > >>
>> > >>> Should we disallow non-string partition columns completely ?
>> > >>> Does anyone depend on that ?
>> > >>>
>> > >>>
>> > >>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>> > >>>
>> > >>> >
>> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>> > >>> >
>> > >>> >The patch above broke drop partitions if the partition happens to be
>> > >>> >non-string.
>> > >>> >This is due to a JDO issue with non-string columns.
>> > >>> >
>> > >>> >Is anyone using non-string partition columns ?
>> > >>> >Should be force the partition columns to be only of type string ?
>> > >>> >The documentation probably does not specify anything clearly.
>> > >>> >
>> > >>> >If someone is dependent on non-string partition column, we need to
>> > >>>revert
>> > >>> >this patch, or make a
>> > >>> >special case for string partition columns.
>> > >>> >
>> > >>> >Thanks,
>> > >>> >-namit
>> > >>> >
>> > >>> >
>> > >>>
>> > >>>
>> > >
>> >
>> >
>>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"

OK, I will keep the support.
Add special casing for string columns in DDLTask

On 5/29/12 9:27 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:

>We use them to we store our dates as integers like 20120130. This
>allows us to do partition pruning with ranges.
>
>On Tue, May 29, 2012 at 4:10 AM, Aniket Mokashi <[EMAIL PROTECTED]>
>wrote:
>> We are using non-string partition columns in production as well.
>>
>> Thanks,
>> Aniket
>>
>> On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
>> <[EMAIL PROTECTED]>wrote:
>>
>>> We're using non-string partition columns in production. I think non
>>>string
>>> partition columns are a good thing to have - it allows you to do all
>>>sorts
>>> of date range calculations etc. AFAIK, MySQL's partition columns can
>>>be of
>>> any type.
>>>
>>> Phil.
>>> On May 26, 2012 7:55 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>>>
>>> > Should I go ahead and file a jira to disallow non-string partition
>>> columns
>>> > ?
>>> > Or, someone depends on that functionality.
>>> >
>>> >
>>> > On 5/25/12 10:01 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>>> >
>>> > >Yes, but the meta-question is:
>>> > >
>>> > >Is anyone dependent on non-string partition columns ? Should we
>>>drop the
>>> > >support for non-string
>>> > >partition columns ?
>>> > >
>>> > >
>>> > >Thanks,
>>> > >-namit
>>> > >
>>> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]>
>>>wrote:
>>> > >
>>> > >>Some discussion for this has happened on
>>> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
>>> > problem
>>> > >>same as the one which I described on that jira ?
>>> > >>
>>> > >>Thanks,
>>> > >>Ashutosh
>>> > >>
>>> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
>>> > >>
>>> > >>> Should we disallow non-string partition columns completely ?
>>> > >>> Does anyone depend on that ?
>>> > >>>
>>> > >>>
>>> > >>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
>>> > >>>
>>> > >>> >
>>> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>>> > >>> >
>>> > >>> >The patch above broke drop partitions if the partition happens
>>>to be
>>> > >>> >non-string.
>>> > >>> >This is due to a JDO issue with non-string columns.
>>> > >>> >
>>> > >>> >Is anyone using non-string partition columns ?
>>> > >>> >Should be force the partition columns to be only of type string
>>>?
>>> > >>> >The documentation probably does not specify anything clearly.
>>> > >>> >
>>> > >>> >If someone is dependent on non-string partition column, we need
>>>to
>>> > >>>revert
>>> > >>> >this patch, or make a
>>> > >>> >special case for string partition columns.
>>> > >>> >
>>> > >>> >Thanks,
>>> > >>> >-namit
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> > >>>
>>> > >
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> "...:::Aniket:::... Quetzalco@tl"

FWIW.. HCatalog only allows partition columns of type string precisely
because in backend datastore type information is not recorded. In my
opinion, partition type should be restricted to type string until we fix
this problem, otherwise it gives unexpected behavior to endusers and/or
bug-reps. One possibility is to introduce config variable
hive.partition.column.type and has it value set to "string" by default.
This ensures that new users get expected behavior of string-only partition
columns. Users who already use other types can reset this config value to
"all" in their deployment when they upgrade to newer version of Hive
(assuming new version comes out without a proper fix). This extra step of
reseting default config will help them to understand the risk they are
taking by changing default value.

Thanks,
Ashutosh

On Tue, May 29, 2012 at 10:02 AM, Namit Jain <[EMAIL PROTECTED]> wrote:

> OK, I will keep the support.
> Add special casing for string columns in DDLTask
>
> On 5/29/12 9:27 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:
>
> >We use them to we store our dates as integers like 20120130. This
> >allows us to do partition pruning with ranges.
> >
> >On Tue, May 29, 2012 at 4:10 AM, Aniket Mokashi <[EMAIL PROTECTED]>
> >wrote:
> >> We are using non-string partition columns in production as well.
> >>
> >> Thanks,
> >> Aniket
> >>
> >> On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
> >> <[EMAIL PROTECTED]>wrote:
> >>
> >>> We're using non-string partition columns in production. I think non
> >>>string
> >>> partition columns are a good thing to have - it allows you to do all
> >>>sorts
> >>> of date range calculations etc. AFAIK, MySQL's partition columns can
> >>>be of
> >>> any type.
> >>>
> >>> Phil.
> >>> On May 26, 2012 7:55 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >>>
> >>> > Should I go ahead and file a jira to disallow non-string partition
> >>> columns
> >>> > ?
> >>> > Or, someone depends on that functionality.
> >>> >
> >>> >
> >>> > On 5/25/12 10:01 AM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >>> >
> >>> > >Yes, but the meta-question is:
> >>> > >
> >>> > >Is anyone dependent on non-string partition columns ? Should we
> >>>drop the
> >>> > >support for non-string
> >>> > >partition columns ?
> >>> > >
> >>> > >
> >>> > >Thanks,
> >>> > >-namit
> >>> > >
> >>> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <[EMAIL PROTECTED]>
> >>>wrote:
> >>> > >
> >>> > >>Some discussion for this has happened on
> >>> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
> >>> > problem
> >>> > >>same as the one which I described on that jira ?
> >>> > >>
> >>> > >>Thanks,
> >>> > >>Ashutosh
> >>> > >>
> >>> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <[EMAIL PROTECTED]> wrote:
> >>> > >>
> >>> > >>> Should we disallow non-string partition columns completely ?
> >>> > >>> Does anyone depend on that ?
> >>> > >>>
> >>> > >>>
> >>> > >>> On 5/24/12 6:49 PM, "Namit Jain" <[EMAIL PROTECTED]> wrote:
> >>> > >>>
> >>> > >>> >
> >>> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> >>> > >>> >
> >>> > >>> >The patch above broke drop partitions if the partition happens
> >>>to be
> >>> > >>> >non-string.
> >>> > >>> >This is due to a JDO issue with non-string columns.
> >>> > >>> >
> >>> > >>> >Is anyone using non-string partition columns ?
> >>> > >>> >Should be force the partition columns to be only of type string
> >>>?
> >>> > >>> >The documentation probably does not specify anything clearly.
> >>> > >>> >
> >>> > >>> >If someone is dependent on non-string partition column, we need
> >>>to
> >>> > >>>revert
> >>> > >>> >this patch, or make a
> >>> > >>> >special case for string partition columns.
> >>> > >>> >
> >>> > >>> >Thanks,
> >>> > >>> >-namit
> >>> > >>> >
> >>> > >>> >
> >>> > >>>
> >>> > >>>
> >>> > >
> >>> >
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> "...:::Aniket:::... Quetzalco@tl"
>
>