Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Using LoadFunc to get arbitrary data into Pig script

Thomas Schlosser 2012-09-07, 13:34
Copy link to this message
Re: Using LoadFunc to get arbitrary data into Pig script
Hi Thomas,
This isn't a complete answer, but take a look at mock.Storage that
Julien wrote to make testing easy:



On Fri, Sep 7, 2012 at 6:34 AM, Thomas Schlosser
> Hi all,
> does anybody know what the easiest/best way is to get the result of an
> arbitrary calculation, which is done in Java, into a Pig relation?
> At the end, I think I would like to have something like:
> A = LOAD '' USING CustomCalculationLoadFunc('$paramA', '$paramB');
> Note: Input file/directory is empty, since the input is configurable via
> external properties.
> Since I have no distributed input, e.g., in the HDFS, the resulting Job
> should -- in my opinion -- run as a single map task.
> Is there another mechanism in Pig or any special InputFormat that can be
> used?
> I also had the problem that there is a mechanism that does not allow to have
> empty locations passed to the LoadFunc.
> Any suggestions how to handle this?
> Thanks in advance.
> Regards,
> Thomas