-Re: Using LoadFunc to get arbitrary data into Pig script
Dmitriy Ryaboy 2012-09-07, 16:56
This isn't a complete answer, but take a look at mock.Storage that
Julien wrote to make testing easy:
On Fri, Sep 7, 2012 at 6:34 AM, Thomas Schlosser
<[EMAIL PROTECTED]> wrote:
> Hi all,
> does anybody know what the easiest/best way is to get the result of an
> arbitrary calculation, which is done in Java, into a Pig relation?
> At the end, I think I would like to have something like:
> A = LOAD '' USING CustomCalculationLoadFunc('$paramA', '$paramB');
> Note: Input file/directory is empty, since the input is configurable via
> external properties.
> Since I have no distributed input, e.g., in the HDFS, the resulting Job
> should -- in my opinion -- run as a single map task.
> Is there another mechanism in Pig or any special InputFormat that can be
> I also had the problem that there is a mechanism that does not allow to have
> empty locations passed to the LoadFunc.
> Any suggestions how to handle this?
> Thanks in advance.