Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> A major addition to Pig. Working with spatial data

Copy link to this message
Re: A major addition to Pig. Working with spatial data
I've just created a new JIRA issue for the spatial functionality.
This issue is all about the new datatype which is the only thing that needs
to be changed internally in Pig in this phase. Pigeon is already working
with the ESRI library but it converts between binary representation and
Geometry class back and forth. Once the new datatype is added, we can
change Pigeon to work with this datatype too. We can still keep the current
conversion functionality as it allows the system to automatically perform
the conversion from the bytearray datatype as it adds the autodetect
functionality when a column is not given a type in the schema.

I don't know if I should provide a patch to this issue myself or there is
someone else who can work on it. I can of course do it but I think it will
take me some time to finish as I'm not yet familiar with the internals of
Pig. Someone who is familiar with the parser would definitely make a better
job here. I can focus on Pigeon and add more spatial functions there so
that we can have a plenty of functions once the new datatype is added. I'm
open to both solutions but I'm just checking with you.


Best regards,
Ahmed Eldawy
On Wed, May 29, 2013 at 12:17 PM, Russell Jurney

> Awesome. This would be a great addition to Pig. Please create a JIRA.
> Russell Jurney http://datasyndrome.com
> On May 29, 2013, at 8:51 AM, Ahmed Eldawy <[EMAIL PROTECTED]> wrote:
> > Hi all,
> >
> > Nick has pointed out to me an alternative GIS package that can replace
> JTS.
> > ESRI has recently released a GIS
> > package<https://github.com/Esri/geometry-api-java>under Apache
> > license. I changed Pigeon to work with that new package. I
> > think it could be easier now to integrate this work with main branch of
> > Apache Pig. I will go on with the current project and add more spatial
> > functionality. We can then add a new datatype to Apache and link it to
> > those functions.
> >
> > ESRI package contains a class OGCGeometry
> > <
> http://esri.github.io/geometry-api-java/javadoc/com/esri/core/geometry/ogc/OGCGeometry.html
> >which
> > can be linked to a new datatype 'Geometry'. Do you think we can rely on
> the
> > new package and integrate the work with Apache Pig?
> >
> > On May 23, 2013 11:40 PM, "Ahmed Eldawy" <[EMAIL PROTECTED]> wrote:
> >
> >> Hi all,
> >>  Thanks for your help. I've started the project with a minimal
> >> functionality as a start. It's currently hosted in github. It is
> licensed
> >> under the Apache public license to make it easier to merge with Pig.
> >> Currently it has only a very few functions. I implemented a function
> from
> >> different types of functions (e.g., Aggregate and create). I'll keep
> adding
> >> functions and any contributions to the project are welcome. As a
> beginning,
> >> I need an ANT build file that runs the tests, compiles and generates a
> jar
> >> file. I'm not familiar with ANT so any help in this is encouraged.
> >> Here's the project home page
> >> https://github.com/aseldawy/pigeon
> >>
> >>
> >> If you have any comments or suggestion please contact me.
> >>
> >>
> >> Best regards,
> >> Ahmed Eldawy
> >>
> >>
> >> On Mon, May 6, 2013 at 3:09 PM, Jonathan Coveney <[EMAIL PROTECTED]
> >wrote:
> >>
> >>> Nick: the only issue is that the way types are implemented in Pig don't
> >>> allow us to easily "plug-in" types externally. Adding support for that
> >>> would be cool, but a fair bit of work.
> >>>
> >>>
> >>> 2013/5/6 Nick Dimiduk <[EMAIL PROTECTED]>
> >>>
> >>>> I'm to a lawyer, but I see no reason why this cannot be an external
> >>>> extension to Pig. It would behave the same way PostGIS is an external
> >>>> extension to Postgres. Any Apache issues would be toward general
> >>>> purpose enhancements, not specific to your project.
> >>>>
> >>>> Good on you!
> >>>> -n
> >>>>
> >>>> On Mon, May 6, 2013 at 10:12 AM, Ahmed Eldawy <[EMAIL PROTECTED]>
> >>> wrote: