Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - A major addition to Pig. Working with spatial data


Copy link to this message
-
Re: A major addition to Pig. Working with spatial data
Ahmed Eldawy 2013-05-05, 23:44
Thanks all for taking the time to respond. Danial, I didn't know that Solr
uses JTS. This is a good finding and we can definitely ask them to see if
there is a work around we can do. Jonathan, I thought of the same idea of
serializing/deserializing a bytearray each time a UDF is called. The
deserialization part is good for letting Pig auto detect spatial types if
not set explicitly in the schema. What is the best way to start this? I
want to add an initial set of JIRA issues and start working on them but I
also need to keep the work grouped in some sense just for organization.

Thanks
Ahmed

Best regards,
Ahmed Eldawy
On Sat, May 4, 2013 at 4:47 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:

> I agree that this is cool, and if other projects are using JTS it is worth
> talking them to see how. I also agree that licensing is very frustrating.
>
> In the short term, however, while it is annoying to have to manage the
> serialization and deserialization yourself, you can have the geometry type
> be passed around as a bytearray type. Your UDF's will have to know this and
> treat it accordingly, but if you did this then all of the tools could be in
> an external project on github instead of a branch in Pig. Then, if we can
> get the licensing done, we could add the Geometry type to Pig. Adding
> types, honestly, is kind of tedious but not super difficult, so once the
> rest is done, that shouldn't be too difficult.
>
>
> 2013/5/4 Russell Jurney <[EMAIL PROTECTED]>
>
> > If a way could be found, this would be an awesome addition to Pig.
> >
> > Russell Jurney http://datasyndrome.com
> >
> > On May 3, 2013, at 4:09 PM, Daniel Dai <[EMAIL PROTECTED]> wrote:
> >
> > > I am not sure how other Apache projects dealing with it? Seems Solr
> also
> > > has some connector to JTS?
> > >
> > > Thanks,
> > > Daniel
> > >
> > >
> > > On Thu, May 2, 2013 at 11:59 AM, Ahmed Eldawy <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> Thanks Alan for your interest. It's too bad that an open source
> > licensing
> > >> issue is holding me back from doing some open source work. I
> understand
> > the
> > >> issue and your workarounds make sense. However, as I mentioned in the
> > >> beginning, I don't want to have my own branch of Pig because it makes
> my
> > >> extension less portable. I'll think of another way to do it. I'll ask
> > vivid
> > >> solutions if they can double license their code although I think the
> > answer
> > >> will be no. I'll also think of a way to ship my extension as a set of
> > jar
> > >> files without the need to change the core of Pig. This way, it can be
> > >> easily ported to newer versions of Pig.
> > >>
> > >> Thanks
> > >> Ahmed
> > >>
> > >> Best regards,
> > >> Ahmed Eldawy
> > >>
> > >>
> > >> On Thu, May 2, 2013 at 12:33 PM, Alan Gates <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >>> I know this is frustrating, but the different licenses do have
> > different
> > >>> requirements that make it so that Apache can't ship GPL code.  A
> legal
> > >>> explanation is at
> > http://www.apache.org/licenses/GPL-compatibility.htmlFor additional info
> > on the LGPL specific questions see
> > >>> http://www.apache.org/legal/3party.html
> > >>>
> > >>> As far as pulling it in via ivy, the issue isn't so much where the
> code
> > >>> lives as much as what code we are requiring to make Pig work.  If
> > >> something
> > >>> that is [L]GPL is required for Pig it violates Apache rules as
> outlined
> > >>> above.  It also would be a show stopper for a lot of companies that
> > >>> redistribute Pig and that are allergic to GPL software.
> > >>>
> > >>> So, as I said before, if you wanted to continue with that library and
> > >> they
> > >>> are not willing to relicense it then it would have to be bolted on
> > after
> > >>> Apache Pig is built.  Nothing stops you from doing this by
> downloading
> > >>> Apache Pig, adding this library and your code, and redistributing,
> > though
> > >>> it wouldn't then be open to all Pig users.