Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Using LoadFunc to get arbitrary data into Pig script


+
Thomas Schlosser 2012-09-07, 13:34
Copy link to this message
-
Re: Using LoadFunc to get arbitrary data into Pig script
Hi Thomas,
This isn't a complete answer, but take a look at mock.Storage that
Julien wrote to make testing easy:

http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/builtin/mock/Storage.java

D

On Fri, Sep 7, 2012 at 6:34 AM, Thomas Schlosser
<[EMAIL PROTECTED]> wrote:
> Hi all,
> does anybody know what the easiest/best way is to get the result of an
> arbitrary calculation, which is done in Java, into a Pig relation?
> At the end, I think I would like to have something like:
> A = LOAD '' USING CustomCalculationLoadFunc('$paramA', '$paramB');
>
> Note: Input file/directory is empty, since the input is configurable via
> external properties.
>
> Since I have no distributed input, e.g., in the HDFS, the resulting Job
> should -- in my opinion -- run as a single map task.
> Is there another mechanism in Pig or any special InputFormat that can be
> used?
> I also had the problem that there is a mechanism that does not allow to have
> empty locations passed to the LoadFunc.
> Any suggestions how to handle this?
>
> Thanks in advance.
>
> Regards,
> Thomas
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB