Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - A major addition to Pig. Working with spatial data


+
Ahmed Eldawy 2013-05-02, 00:40
+
Alan Gates 2013-05-02, 00:50
+
Ahmed Eldawy 2013-05-02, 01:08
+
Alan Gates 2013-05-02, 17:33
+
Ahmed Eldawy 2013-05-02, 18:59
+
Daniel Dai 2013-05-03, 23:09
+
Russell Jurney 2013-05-04, 09:14
+
Jonathan Coveney 2013-05-04, 21:47
+
Ahmed Eldawy 2013-05-05, 23:44
+
Jonathan Coveney 2013-05-06, 14:53
+
Ahmed Eldawy 2013-05-06, 17:12
+
Nick Dimiduk 2013-05-06, 20:03
+
Jonathan Coveney 2013-05-06, 20:09
+
Ahmed Eldawy 2013-05-24, 04:40
Copy link to this message
-
Re: A major addition to Pig. Working with spatial data
Ahmed Eldawy 2013-05-29, 15:50
Hi all,

Nick has pointed out to me an alternative GIS package that can replace JTS.
ESRI has recently released a GIS
package<https://github.com/Esri/geometry-api-java>under Apache
license. I changed Pigeon to work with that new package. I
think it could be easier now to integrate this work with main branch of
Apache Pig. I will go on with the current project and add more spatial
functionality. We can then add a new datatype to Apache and link it to
those functions.

ESRI package contains a class OGCGeometry
<http://esri.github.io/geometry-api-java/javadoc/com/esri/core/geometry/ogc/OGCGeometry.html>which
can be linked to a new datatype 'Geometry'. Do you think we can rely on the
new package and integrate the work with Apache Pig?

On May 23, 2013 11:40 PM, "Ahmed Eldawy" <[EMAIL PROTECTED]> wrote:

> Hi all,
>   Thanks for your help. I've started the project with a minimal
> functionality as a start. It's currently hosted in github. It is licensed
> under the Apache public license to make it easier to merge with Pig.
> Currently it has only a very few functions. I implemented a function from
> different types of functions (e.g., Aggregate and create). I'll keep adding
> functions and any contributions to the project are welcome. As a beginning,
> I need an ANT build file that runs the tests, compiles and generates a jar
> file. I'm not familiar with ANT so any help in this is encouraged.
> Here's the project home page
> https://github.com/aseldawy/pigeon
>
>
> If you have any comments or suggestion please contact me.
>
>
> Best regards,
> Ahmed Eldawy
>
>
> On Mon, May 6, 2013 at 3:09 PM, Jonathan Coveney <[EMAIL PROTECTED]>wrote:
>
>> Nick: the only issue is that the way types are implemented in Pig don't
>> allow us to easily "plug-in" types externally. Adding support for that
>> would be cool, but a fair bit of work.
>>
>>
>> 2013/5/6 Nick Dimiduk <[EMAIL PROTECTED]>
>>
>> > I'm to a lawyer, but I see no reason why this cannot be an external
>> > extension to Pig. It would behave the same way PostGIS is an external
>> > extension to Postgres. Any Apache issues would be toward general
>> > purpose enhancements, not specific to your project.
>> >
>> > Good on you!
>> > -n
>> >
>> > On Mon, May 6, 2013 at 10:12 AM, Ahmed Eldawy <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > > I contacted solr developers to see how JTS can be included in an
>> Apache
>> > > project. See
>> > >
>> > >
>> >
>> http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/raw/%3C1367815102914-4060969.post%40n3.nabble.com%3E/
>> > > As far as I understand, they did not include it in the main solr
>> project,
>> > > rather, they created a separate project (spatial 4j) which is still
>> > > licensed under Apache license and refers to JTS. Users will have to
>> > > download JTS libraries separately to make it run. That's pretty much
>> the
>> > > same plan that Jonathan mentioned. We will still have the overhead of
>> > > serializing/deserializing the shapes each time a function is called.
>> > Also,
>> > > we will have to use the ugly bytearray data type for spatial data
>> instead
>> > > of creating its own data type (e.g., Geometry).
>> > > I think using spatial 4j instead of JTS will not be sufficient for our
>> > case
>> > > as we need to provide an access to all spatial functions of JTS such
>> as
>> > > Union, Intersection, Difference, ... etc. This way we can claim
>> > conformity
>> > > with OGC standards which gives visibility and appreciations of the
>> > spatial
>> > > community.
>> > > I think also that this means I will not add any issues to JIRA as it
>> is
>> > now
>> > > a separate project. I'm planning to host it on github and have all the
>> > > issues there.
>> > > Let me know if you have any suggestions or comments.
>> > >
>> > > Thanks
>> > > Ahmed
>> > >
>> > >
>> > > Best regards,
>> > > Ahmed Eldawy
>> > >
>> > >
>> > > On Mon, May 6, 2013 at 9:53 AM, Jonathan Coveney <[EMAIL PROTECTED]>
>> > > wrote:
>> > >
>> > > > You can give them all the same label or tag and filter on that later
+
Russell Jurney 2013-05-29, 17:17
+
Ahmed Eldawy 2013-06-03, 17:26
+
Russell Jurney 2013-06-03, 17:52