Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Next Pig release proposal


Copy link to this message
-
Re: Next Pig release proposal
Gianmarco De Francisci Mo... 2011-10-22, 09:31
Hi,

just my 2 cents.

I think the issue here is not 1.0 vs 0.10, but what's the versioning scheme
we want to use for Pig.
Up to now it has been just an increasing number after a '0.' prefix, changed
when the community felt it was time. I think this works well for a small
project, but it is somewhat fuzzy.

I like the idea of having <major>.<minor>.<patch> versions like many other
projects. It's a very clear and almost standard way of versioning a piece of
software. It has clear rules on when to change each of the numbers, and lets
the user get an idea of backward compatibility at a glance.

So, to conclude, I am in favor of going 1.0 (or 1.0.0) as long as we decide
a clear versioning policy (whichever it is).
So that the 1.0 milestone would mark the beginning of our new policy.

Cheers,
--
Gianmarco

On Fri, Oct 21, 2011 at 23:10, <[EMAIL PROTECTED]> wrote:

> If one were to rewrite input and output formats to use the webhdfs://
> APIs, this would not be an issue, right ?
>
> - milind
>
>
> On 10/21/11 1:50 PM, "Santhosh Srinivasan" <[EMAIL PROTECTED]> wrote:
>
> >If I was not clear in my earlier email, I apologize for the lack of
> >clarity. I am no longer in favour of waiting for Hadoop API stability
> >across Hadoop versions. It's a pipe dream.
> >
> >When we had PigInputFormat and PigOutputFormat, your reasoning would be
> >spot on. I am concerned about the following. Our tight integration with
> >Hadoop due to the use of Input and Output format might lead to a break in
> >backward compatibility. I am not sure if the comparison with that of Java
> >is valid. Probably a majority of the users don't use JNI. Its very hard
> >to use Pig without writing custom load and store functions. The default
> >load and store don't suffice for a majority of use cases that I have
> >observed.
> >
> >I am trying to get all factors that might influence this decision. From
> >the few emails that have been exchanged since yesterday, we have the
> >following factors:
> >
> >1. Hadoop 0.20.205 (support for Append)
> >2. Hadoop 0.22
> >3. Hadoop 0.23
> >4. Maturity of the new parser
> >5. Stability of the new logical plan
> >6. Other components in the eco system.
> >       - Avro (1.5.4, 1.4.1, ...)
> >       - Cassandra (1.0.0, 0.8.7, ...)
> >       - Chukwa (0.4.0, 0.3.0, ...)
> >       - Hama (0.3.0, 0.2.0, ...)
> >       - Hbase (0.90.4, 0.90.3, 0.90.2, 0.90.1, ...)
> >       - Hive (Releases - 0.7.1, 0.7.0, 0.6.0, ...)
> >       - Zookeeper (3.3.3, 3.3.2, 3.2.2, 3.1.2, ...)
> >
> >Santhosh
> >
> >
> >-----Original Message-----
> >From: Thejas Nair [mailto:[EMAIL PROTECTED]]
> >Sent: Friday, October 21, 2011 11:22 AM
> >To: [EMAIL PROTECTED]
> >Subject: Re: Next Pig release proposal
> >
> >
> >Santosh,
> >I thought you meant API stability for hadoop across major versions, but I
> >guess you are referring to stability within 0.23 versions. But argument
> >applies to that as well, if 0.23.1 is not compatible with 0.23.0, we need
> >to call the release for 0.23.1 as 'pig 1.x for 0.23.1 api' .
> >
> >We just need to communicate to the users that the
> >InputFormat/OutputFormat api's (and any anything else we expose from
> >hadoop) depends on the hadoop version they are using.
> >
> >I think it is just like different JNI libraries that you would write for
> >different OS. But the java version remains the same across OSs.
> >
> >-Thejas
> >
> >
> >On 10/21/11 10:59 AM, Santhosh Srinivasan wrote:
> >> Thejas,
> >>
> >> I guess you did not read my email completely. You are referring to the
> >>premise without examining the conclusion. I am repasting my entire email
> >>to avoid confusion (I hate truncated references). If you could respond
> >>again, it will bring us onto the same page.
> >>
> >> <email>
> >>
> >> Ref: http://tinyurl.com/4ng8upa (last discussion on 1.0)
> >>
> >> How far have we progressed from our last discussion in March. There was
> >>no consensus on the 1.0 release. Opinions ranged from having more
> >>releases to bake in the maturity of the new parser and logical plan