Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Using LoadFunc to get arbitrary data into Pig script


Copy link to this message
-
Re: Using LoadFunc to get arbitrary data into Pig script
Hi Thomas,
This isn't a complete answer, but take a look at mock.Storage that
Julien wrote to make testing easy:

http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/builtin/mock/Storage.java

D

On Fri, Sep 7, 2012 at 6:34 AM, Thomas Schlosser
<[EMAIL PROTECTED]> wrote:
> Hi all,
> does anybody know what the easiest/best way is to get the result of an
> arbitrary calculation, which is done in Java, into a Pig relation?
> At the end, I think I would like to have something like:
> A = LOAD '' USING CustomCalculationLoadFunc('$paramA', '$paramB');
>
> Note: Input file/directory is empty, since the input is configurable via
> external properties.
>
> Since I have no distributed input, e.g., in the HDFS, the resulting Job
> should -- in my opinion -- run as a single map task.
> Is there another mechanism in Pig or any special InputFormat that can be
> used?
> I also had the problem that there is a mechanism that does not allow to have
> empty locations passed to the LoadFunc.
> Any suggestions how to handle this?
>
> Thanks in advance.
>
> Regards,
> Thomas