Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> A major addition to Pig. Working with spatial data

Ahmed Eldawy 2013-05-02, 00:40
Copy link to this message
Re: A major addition to Pig. Working with spatial data
Passing on the technical details for a moment, I see a licensing issue.  JTS is licensed under LGPL.  Apache projects cannot contain or ship [L]GPL.  Apache does not meet the requirements of GPL and thus we cannot repackage their code. If you wanted to go forward using that class this would have to be packaged as an add on that was downloaded separately and not from Apache.  Another option is to work with the JTS community and see if they are willing to dual license their code under BSD or Apache license so that Pig could include it.  If neither of those are an option you would need to come up with a new class to contain your spatial data.


On May 1, 2013, at 5:40 PM, Ahmed Eldawy wrote:

> Hi all,
>  First, sorry for the long email. I wanted to put all my thoughts here and
> get your feedback.
>  I'm proposing a major addition to Pig that will greatly increase its
> functionality and user base. It is simply to add spatial support to the
> language and the framework. I've already started working on that but I
> don't want it to be just another branch. I want it, eventually, to be
> merged with the trunk of Apache Pig. So, I'm sending this email mainly to
> reach out the main contributors of Pig to see the feasibility of this.
> This addition is a part of a big project we have been working on in
> University of Minnesota; the project is called Spatial Hadoop.
> http://spatialhadoop.cs.umn.edu. It's about building a MapReduce framework
> (Hadoop) that is capable of maintaining and analyzing spatial data
> efficiently. I'm the main guy behind that project and since we released its
> first version, we received very encouraging responses from different groups
> in the research and industrial community. I'm sure the addition we want to
> make to Pig Latin will be widely accepted by the people in the spatial
> community.
> I'm proposing a plan here while we're still in the early phases of this
> task to be able to discuss it with the main contributors and see its
> feasibility. First of all, I think that we need to change the core of Pig
> to be able to support spatial data. Providing a set of UDFs only is not
> enough. The main reason is that Pig Latin does not provide a way to create
> a new data type which is needed for spatial data. Once we have the spatial
> data types we need, the functionality can be expanded using more UDFs.
> Here's the plan as I see it.
> 1- Introduce a new primitive data type Geometry which represents all
> spatial data types. In the underlying system, this will map to
> com.vividsolutions.jts.geom.Geometry. This is a class from Java Topology
> Suite (JTS) [http://www.vividsolutions.com/jts/JTSHome.htm], a stable and
> efficient open source Java library for spatial data types and algorithms.
> It is very popular in the spatial community and a C++ port of it is used in
> PostGIS [http://postgis.net/] (a spatial library for Postgres). JTS also
> conforms with Open Geospatial Consortium (OGC) [
> http://www.opengeospatial.org/] which is an open standard for the spatial
> data types. The Geometry data type is read from and written to text files
> using the Well Known Text (WKT) format. There is also a way to convert it
> to/from binary so that it can work with binary files and streams.
> 2- Add functions that manipulate spatial data types. These will be added as
> UDFs and we will not need to mess with the internals of Pig. Most probably,
> there will be one new class for each operation (e.g., union or
> intersection). I think it will be good to put these new operations inside
> the core of Pig so that users can use it without having to write the fully
> qualified class name. Also, since there is no way to implicitly cast a
> spatial data type to a non-spatial data types, there will not be any
> conflicts in existing operations or new operations. All new operations, and
> only the new operations, will be working on spatial data types. Here is an
> initial list of operations that can be added. All those operations are
Ahmed Eldawy 2013-05-02, 01:08
Alan Gates 2013-05-02, 17:33
Ahmed Eldawy 2013-05-02, 18:59
Daniel Dai 2013-05-03, 23:09
Russell Jurney 2013-05-04, 09:14
Jonathan Coveney 2013-05-04, 21:47
Ahmed Eldawy 2013-05-05, 23:44
Jonathan Coveney 2013-05-06, 14:53
Ahmed Eldawy 2013-05-06, 17:12
Nick Dimiduk 2013-05-06, 20:03
Jonathan Coveney 2013-05-06, 20:09
Ahmed Eldawy 2013-05-24, 04:40
Ahmed Eldawy 2013-05-29, 15:50
Russell Jurney 2013-05-29, 17:17
Ahmed Eldawy 2013-06-03, 17:26
Russell Jurney 2013-06-03, 17:52